Envelope encoding of speech signals for transmission to cutaneous actuators

Patent metadata

Field	Value
Publication number	US-10943503-B2
Application number	US-201815949392-A
Country	US
Kind code	B2
Filing date	Apr 10, 2018
Priority date	Apr 17, 2017
Publication date	Mar 9, 2021
Grant date	Mar 9, 2021

Abstract

A haptic communication device includes a speech signal generator configured to receive speech sounds or a textual message and generate speech signals corresponding to the speech sounds or the textual message. An envelope encoder is operably coupled to the speech signal generator to extract a temporal envelope from the speech signals. The temporal envelope represents changes in amplitude of the speech signals. Carrier signals having a periodic waveform are generated. Actuator signals are generated by encoding the changes in the amplitude of the speech signals from the temporal envelope into the carrier signals. One or more cutaneous actuators are operably coupled to the envelope encoder to generate haptic vibrations representing the speech sounds or the textual message using the actuator signals.

First claim

What is claimed is:

1. A haptic communication system, comprising:
a speech signal generator configured to receive speech sounds or a textual message and generate speech signals corresponding to the speech sounds or the textual message;
an envelope encoder operably coupled to the speech signal generator and configured to:
extract a temporal envelope from the speech signals, the temporal envelope representing changes in amplitude of the speech signals,
generate carrier signals having a periodic waveform,
generate haptic cues by processing the speech signals, and
generate actuator signals by embedding the changes in the amplitude of the speech signals from the temporal envelope into the carrier signals; and

cutaneous actuators operably coupled to the envelope encoder and configured to generate haptic vibrations representing the speech sounds or the textual message using the actuator signals, a number of the cutaneous actuators activated determined by the haptic cues.

2. The haptic communication system of claim 1 , wherein the speech signal generator comprises at least one of:
a microphone configured to generate the speech signals by digitizing the speech sounds; or
a speech synthesizer configured to generate the speech signals by mapping the textual message to stored sound signals.

3. The haptic communication system of claim 1 , wherein the envelope encoder comprises:
a rectifier configured to generate rectified signals by rectifying the speech signals; and
a low pass filter configured to generate the temporal envelope by filtering out frequency components of the rectified signals that are higher than a threshold frequency.

4. The haptic communication system of claim 1 , wherein the envelope encoder extracts the temporal envelope from the speech signals by performing a Hilbert transformation on the speech signals.

5. The haptic communication system of claim 1 , wherein the carrier signals comprise a sinusoidal wave or a square wave train.

6. The haptic communication system of claim 1 , wherein the envelope encoder is configured to generate the actuator signals by:
determining the amplitude of the speech signals from the temporal envelope; and
increasing or decreasing an amplitude of the carrier signals proportional to the amplitude of the speech signals.

7. The haptic communication system of claim 1 , wherein the envelope encoder is configured to generate the actuator signals by:
determining the amplitude of the speech signals from the temporal envelope; and
increasing or decreasing a frequency of the carrier signals proportional to the amplitude of the speech signals.

8. The haptic communication system of claim 1 , wherein the envelope encoder is configured to generate the actuator signals by:
determining the amplitude of the speech signals from the temporal envelope; and
altering the carrier signals to increase or decrease a number of cutaneous actuators activated by the actuator signals, the number proportional to the amplitude of the speech signals.

9. The haptic communication system of claim 1 , wherein the envelope encoder is configured to extract the temporal envelope from the speech signals by:
decomposing the speech signals into a plurality of frequency bands; and
extracting a corresponding temporal envelope from each frequency band, each corresponding temporal envelope representing changes in amplitude of the frequency band, the envelope encoder configured to generate the actuator signals by encoding changes in the amplitude of the frequency band from each corresponding temporal envelope into the carrier signals.

10. The haptic communication system of claim 1 , wherein the envelope encoder comprises:
a low pass filter configured to generate low frequency components of the speech signals by filtering out frequency components of the speech signals that are higher than a threshold frequency; and
a high pass filter configured to generate high frequency components of the speech signals by filtering out frequency components of the speech signals that are lower than the threshold frequency, wherein the envelope encoder further configured to:
determine a low frequency signal power of the low frequency components;
determine a high frequency signal power of the high frequency components;
determine a ratio of the high frequency signal power to the low frequency signal power, and
generate the actuator signals by encoding the ratio of the high frequency signal power to the low frequency signal power into the carrier signals.

11. The haptic communication system of claim 1 , wherein the envelope encoder comprises a machine learning engine configured to:
extract a feature vector from the speech signals, the feature vector representing amplitudes of a plurality of frequency bands of the speech signals, wherein the envelope encoder is further configured to decompose the speech signals into the plurality of frequency bands; and
generate haptic cues representing the speech sounds or the textual message by a machine learning model using the feature vector, the envelope encoder further configured to generate the actuator signals by encoding the haptic cues into the carrier signals.

12. A method, comprising:
receiving speech sounds or a textual message;
generating speech signals corresponding to the speech sounds or the textual message;
extracting a temporal envelope from the speech signals, the temporal envelope representing changes in amplitude of the speech signals;
generating carrier signals having a periodic waveform;
generating haptic cues by processing the speech signals;
generating actuator signals by embedding the changes in the amplitude of the speech signals from the temporal envelope into the carrier signals; and
generating, by cutaneous actuators, haptic vibrations representing the speech sounds or the textual message using the actuator signals, a number of the cutaneous actuators activated determined by the haptic cues.

13. The method of claim 12 , wherein the generating of the speech signals comprises at least one of:
digitizing the speech sounds by a microphone; or
mapping the textual message to stored sound signals by a speech synthesizer.

14. The method of claim 12 , wherein the extracting of the temporal envelope comprises:
generating rectified signals by rectifying the speech signals; and
generating the temporal envelope by filtering out high frequency components of the rectified signals that are higher than a threshold frequency.

15. The method of claim 12 , wherein the generating of the actuator signals comprises:
determining the amplitude of the speech signals from the temporal envelope; and
increasing or decreasing an amplitude or a frequency of the carrier signals proportional to the amplitude of the speech signals.

16. The method of claim 12 , wherein the generating of the actuator signals comprises:
determining the amplitude of the speech signals from the temporal envelope; and
altering the carrier signals to increase or decrease a number of the cutaneous actuators activated by the actuator signals, the number proportional to the amplitude of the speech signals.

17. The method of claim 12 , wherein the extracting of the temporal envelope comprises:
decomposing the speech signals into a plurality of frequency bands; and
extracting a corresponding temporal envelope from each frequency band, each corresponding temporal envelope representing changes in amplitude of the frequency band, the generating of the actuator signals comprising encoding changes in the amplitude of the frequency band from each

Assignees

• Facebook Inc

Inventors

• Turcott Robert
• Israr Ali
• Knott Brian Alexander
• Abnousi Freddy

Classifications

• G06N3/045

  Combinations of networks · CPC title

• G06N3/044

  Recurrent networks, e.g. Hopfield networks · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06N3/0495

  Quantised networks; Sparse networks; Compressed networks · CPC title