Neural network model for generation of compressed haptic actuator signal from audio input

Patent metadata

Field	Value
Publication number	US-2018300651-A1
Application number	US-201815949425-A
Country	US
Kind code	A1
Filing date	Apr 10, 2018
Priority date	Apr 17, 2017
Publication date	Oct 18, 2018
Grant date	—

Abstract

A method comprises inputting an audio signal into a machine learning circuit to compress the audio signal into a sequence of actuator signals. The machine learning circuit being trained by: receiving a training set of acoustic signals and pre-processing the training set of acoustic signals into pre-processed audio data. The pre-processed audio data including at least a spectrogram. The training further includes training the machine learning circuit using the pre-processed audio data. The neural network has a cost function based on a reconstruction error and a plurality of constraints. The machine learning circuit generates a sequence of haptic cues corresponding to the audio input. The sequence of haptic cues is transmitted to a plurality of cutaneous actuators to generate a sequence of haptic outputs.

First claim

What is claimed is:

1 . A method, comprising:
inputting an audio signal into a machine learning circuit to compress the audio signal into a sequence of actuator signals, the machine learning circuit being trained by:
receiving a training set of acoustic signals;
pre-processing the training set of acoustic signals into pre-processed audio data, the pre-processed audio data including at least a spectrogram;
training the machine learning circuit using the pre-processed audio data, the machine learning circuit having a cost function based on a reconstruction error and a plurality of constraints, the machine learning circuit generating a sequence of haptic cues corresponding to the audio input; and
wherein the sequence of haptic cues is transmitted to one or more cutaneous actuators to generate a sequence of haptic outputs.

2 . The method of claim 1 , wherein the spectrogram represents the magnitude as a log amplitude.

3 . The method of claim 1 , wherein the reconstruction error is generated by determining a difference between the pre-processed audio data and an output from the machine learning circuit trained to reconstruct the pre-processed audio data using the sequence of haptic cues generated from the machine learning circuit.

4 . The method of claim 1 , wherein the set of constraints includes a restriction on a number of haptic cues generated by the machine learning circuit for a particular time slice.

5 . The method of claim 1 , wherein the set of constraints includes a constraint that increases a value of the cost function in proportion to a number of activated cutaneous actuators indicated by the haptic cues generated by the machine learning circuit for a particular time slice.

6 . The method of claim 1 , wherein the set of constraints includes a constraint that increases a value of the cost function in proportion to a number state changes of cutaneous actuators as indicated by the haptic cues generated by the machine learning circuit between time slices.

7 . The method of claim 1 , wherein the set of constraints includes a constraint that increases a value of the cost function when the haptic cues generated by the machine learning circuit for a time slice indicates activation of cutaneous actuators from different pre-specified groups of cutaneous actuators.

8 . A set of coefficients for a machine learning algorithm that is generated by:
receiving a training set of acoustic signals; pre-processing the training set of acoustic signals into pre-processed audio data, the pre-processed audio data including at least a spectrogram; training the machine learning algorithm using the pre-processed audio data, the machine learning algorithm having a cost function based on a reconstruction error and a plurality of constraints, the machine learning algorithm generating a sequence of haptic cues corresponding to the audio input.

9 . The set of coefficients of claim 8 , wherein the spectrogram represents the magnitude as a log amplitude.

10 . The set of coefficients of claim 8 , wherein the reconstruction error is generated by determining a difference between the pre-processed audio data and an output from the machine learning algorithm trained to reconstruct the pre-processed audio data using the sequence of haptic cues generated from the machine learning algorithm.

11 . The set of coefficients of claim 8 , wherein the constraints includes a restriction on the number of haptic cues generated by the machine learning algorithm for a particular time slice.

12 . The set of coefficients of claim 8 , wherein the constraints includes a constraint that increases a value of the cost function in proportion to a number of activated cutaneous actuators indicated by the haptic cues generated by the machine learning algorithm for a particular time slice.

13 . The set of coefficients of claim 8 , wherein the constraints includes a constraint that increases a value of the cost function in proportion to a number state changes of cutaneous actuators as indicated by the haptic cues generated by the machine learning algorithm between time slices.

14 . The set of coefficients of claim 8 , wherein the constraints includes a constraint that increases a value of the cost function when the haptic cues generated by the machine learning algorithm for a time slice indicates activation of cutaneous actuators from different pre-specified groups of cutaneous actuators.

15 . A non-transitory computer readable storage medium, comprising instructions, that when executed by a processor, cause the processor to:
input an audio signal into a machine learning circuit to compress the audio signal into a sequence of actuator signals, the machine learning circuit being trained by:
receiving a training set of acoustic signals;
pre-processing the training set of acoustic signals into pre-processed audio data, the pre-processed audio data including at least a spectrogram;
training the machine learning circuit using the pre-processed audio data, the machine learning circuit having a cost function based on a reconstruction error and a plurality of constraints, the machine learning circuit generating a sequence of haptic cues corresponding to the audio input; and
wherein the sequence of haptic cues is transmitted to one or more cutaneous actuators to generate a sequence of haptic outputs, the one or more cutaneous actuators facing a skin surface of a user's body.

16 . The non-transitory computer readable storage medium of claim 15 , wherein the reconstruction error is generated by determining a difference between the pre-processed audio data and an output from the machine learning circuit trained to reconstruct the pre-processed audio data using the sequence of haptic cues generated from the machine learning circuit.

17 . The non-transitory computer readable storage medium of claim 15 , wherein the constraints includes a restriction on the number of haptic cues generated by the machine learning circuit for a particular time slice.

18 . The non-transitory computer readable storage medium of claim 15 , wherein the constraints includes a constraint that increases a value of the cost function in proportion to a number of activated cutaneous actuators indicated by the haptic cues generated by the machine learning circuit for a particular time slice.

19 . The non-transitory computer readable storage medium of claim 15 , wherein the constraints includes a constraint that increases a value of the cost function in proportion to a number state changes of cutaneous actuators as indicated by the haptic cues generated by the machine learning circuit between time slices.

20 . The non-transitory computer readable storage medium of claim 15 , wherein the constraints includes a constraint that increases a value of the cost function when the haptic cues generated by the machine learning circuit for a time slice indicates activation of cutaneous actuators from different pre-specified groups of cutaneous actuators.

Assignees

• Facebook Inc

Inventors

• Knott Brian Alexander
• Chakkabala Venkatasiva Prasad

Classifications

• G06N3/044

  Recurrent networks, e.g. Hopfield networks · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G09B21/004

  Details of particular tactile cells, e.g. electro-mechanical or mechanical layout · CPC title

• G08B6/00

  Tactile signalling systems, e.g. tactile personal calling systems · CPC title

• G09B21/003 Primary

  using tactile presentation of the information, e.g. Braille displays · CPC title