Neural network model for generation of compressed haptic actuator signal from audio input

Patent metadata

Field	Value
Publication number	US-11355033-B2
Application number	US-201815949425-A
Country	US
Kind code	B2
Filing date	Apr 10, 2018
Priority date	Apr 17, 2017
Publication date	Jun 7, 2022
Grant date	Jun 7, 2022

Abstract

A method comprises inputting an audio signal into a machine learning circuit to compress the audio signal into a sequence of actuator signals. The machine learning circuit being trained by: receiving a training set of acoustic signals and pre-processing the training set of acoustic signals into pre-processed audio data. The pre-processed audio data including at least a spectrogram. The training further includes training the machine learning circuit using the pre-processed audio data. The neural network has a cost function based on a reconstruction error and a plurality of constraints. The machine learning circuit generates a sequence of haptic cues corresponding to the audio input. The sequence of haptic cues is transmitted to a plurality of cutaneous actuators to generate a sequence of haptic outputs.

First claim

What is claimed is:

1. A method, comprising:
inputting an audio signal into a machine learning circuit to compress the audio signal into a sequence of actuator signals, the machine learning circuit being trained by:
receiving a training set of acoustic signals;
pre-processing the training set of acoustic signals into a plurality of time slices of pre-processed audio data and converting each of the time slices of the pre-processed audio data into a spectrogram;
training the machine learning circuit using the pre-processed audio data to reduce a value of a cost function based on a reconstruction error and a plurality of constraints, the machine learning circuit generating a sequence of haptic cues corresponding to the audio signal, at least one of the plurality of constraints associated with:
increasing the value of the cost function responsive to an increase in changes of adjacent haptic cues of the sequence corresponding to adjacent ones of the time slices, and
decreasing the value of the cost function responsive to a decrease in the changes of the adjacent haptic cues; and

wherein the sequence of haptic cues is transmitted to one or more cutaneous actuators to generate a sequence of haptic outputs.

2. The method of claim 1 , wherein the spectrogram represents the magnitude as a log amplitude.

3. The method of claim 1 , wherein the reconstruction error is generated by determining a difference between the pre-processed audio data and an output from the machine learning circuit trained to reconstruct the pre-processed audio data using the sequence of haptic cues generated from the machine learning circuit.

4. The method of claim 1 , wherein the plurality of constraints includes a restriction on a number of haptic cues generated by the machine learning circuit for each of the time slices.

5. The method of claim 1 , wherein the plurality of constraints includes a constraint that increases the value of the cost function in proportion to a number of activated cutaneous actuators indicated by the haptic cues generated by the machine learning circuit for a particular time slice.

6. The method of claim 1 , wherein the plurality of constraints includes a constraint that increases the value of the cost function in proportion to a number state changes of cutaneous actuators as indicated by the haptic cues generated by the machine learning circuit between the time slices.

7. The method of claim 1 , wherein the plurality of constraints includes a constraint that increases the value of the cost function when the haptic cues generated by the machine learning circuit for one of the time slices indicates activation of cutaneous actuators from different pre-specified groups of cutaneous actuators.

8. A non-transitory computer readable storage medium, comprising instructions, that when executed by a processor, cause the processor to:
input an audio signal into a machine learning circuit to compress the audio signal into a sequence of actuator signals, the machine learning circuit being trained by:
receiving a training set of acoustic signals;
pre-processing the training set of acoustic signals into a plurality of time slices of pre-processed audio data and converting each of the time slices of the pre-processed audio data into a spectrogram;
training the machine learning circuit using the pre-processed audio data to reduce a value of a cost function based on a reconstruction error and a plurality of constraints, the machine learning circuit generating a sequence of haptic cues corresponding to the audio signal, at least one of the plurality of constraints associated with:
increasing the value of the cost function responsive to an increase in changes of adjacent haptic cues of the sequence corresponding to adjacent ones of the time slices, and
decreasing the value of the cost function responsive to a decrease in the changes of the adjacent haptic cues; and

wherein the sequence of haptic cues is transmitted to one or more cutaneous actuators to generate a sequence of haptic outputs, the one or more cutaneous actuators facing a skin surface of a user's body.

9. The non-transitory computer readable storage medium of claim 8 , wherein the reconstruction error is generated by determining a difference between the pre-processed audio data and an output from the machine learning circuit trained to reconstruct the pre-processed audio data using the sequence of haptic cues generated from the machine learning circuit.

10. The non-transitory computer readable storage medium of claim 8 , wherein the plurality of constraints include a restriction on the number of haptic cues generated by the machine learning circuit for each of the time slices.

11. The non-transitory computer readable storage medium of claim 8 , wherein the plurality of constraints include a constraint that increases the value of the cost function in proportion to a number of activated cutaneous actuators indicated by the haptic cues generated by the machine learning circuit for each of the time slices.

12. The non-transitory computer readable storage medium of claim 8 , wherein the constraints includes a constraint that increases the value of the cost function in proportion to a number state changes of cutaneous actuators as indicated by the haptic cues generated by the machine learning circuit between the time slices.

13. The non-transitory computer readable storage medium of claim 8 , wherein the constraints includes a constraint that increases a value of the cost function when the haptic cues generated by the machine learning circuit for one of the time slices indicates activation of cutaneous actuators from different pre-specified groups of cutaneous actuators.

Assignees

• Meta Platforms Inc

Inventors

• Knott Brian Alexander
• Chakkabala Venkatasiva Prasad

Classifications

• G06N3/045

  Combinations of networks · CPC title

• G06N3/044

  Recurrent networks, e.g. Hopfield networks · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06N3/0495

  Quantised networks; Sparse networks; Compressed networks · CPC title