Context-data based speech enhancement

Patent metadata

Field	Value
Publication number	US-12380909-B2
Application number	US-202318334641-A
Country	US
Kind code	B2
Filing date	Jun 14, 2023
Priority date	Mar 23, 2021
Publication date	Aug 5, 2025
Grant date	Aug 5, 2025

Abstract

A device to perform speech enhancement includes one or more processors configured to process image data to detect at least one of an emotion, a speaker characteristic, or a noise type. The one or more processors are also configured to generate context data based at least in part on the at least one of the emotion, the speaker characteristic, or the noise type. The one or more processors are further configured to obtain input spectral data based on an input signal. The input signal represents sound that includes speech. The one or more processors are also configured to process, using a multi-encoder transformer, the input spectral data and the context data to generate output spectral data that represents a speech enhanced version of the input signal.

First claim

What is claimed is:

1. A device to perform speech enhancement, the device comprising:
one or more processors configured to:
process image data to detect at least one of an emotion, a speaker identification, or a noise type;
generate context data that represents the at least one of the emotion, the speaker identification, or the noise type;
obtain input spectral data based on an input signal that corresponds to the image data, the input signal representing sound that includes speech;
provide the input spectral data to a first encoder of a multi-encoder transformer to generate first encoded data;
provide the context data to at least a second encoder of the multi-encoder transformer to generate second encoded data;
provide the first encoded data and the second encoded data to a decoder of the multi-encoder transformer to generate output spectral data that represents a speech enhanced version of the input signal; and
perform speech synthesis on the output spectral data to generate an output waveform corresponding to an enhanced version of the speech.

2. The device of claim 1 , wherein the multi-encoder transformer includes:
a multi-encoder that includes:
the first encoder, wherein the first encoder includes a first attention network;
the at least a second encoder, wherein the at least a second encoder includes a second attention network; and
the decoder, wherein the decoder includes a decoder attention network.

3. The device of claim 2 , wherein the one or more processors are configured to:
obtain the context data based on one or more data sources; and
provide, to the decoder attention network, the first encoded data and the second encoded data to generate the output spectral data.

4. The device of claim 3 , wherein the one or more data sources includes the input signal and the image data.

5. The device of claim 3 , wherein the decoder attention network comprises:
a first multi-head attention network configured to process the first encoded data;
a second multi-head attention network configured to process the second encoded data; and
a combiner configured to combine outputs of the first multi-head attention network and the second multi-head attention network.

6. The device of claim 2 , wherein the decoder further comprises:
a masked multi-head attention network coupled to an input of the decoder attention network; and
a decoder feed forward network coupled to an output of the decoder attention network.

7. The device of claim 2 , the first encoder including a Mel filter bank configured to filter the input spectral data.

8. The device of claim 2 , further comprising an automatic speech recognition engine configured to generate text based on the input signal, wherein the context data includes the text.

9. The device of claim 8 , wherein the second encoder includes a grapheme-to-phoneme convertor configured to process the text.

10. The device of claim 2 , wherein:
the first encoder comprises:
a first layer including the first attention network, wherein the first attention network corresponds to a first multi-head attention network; and
a second layer including a first feed forward network, and

the second encoder comprises:
a third layer including the second attention network, wherein the second attention network corresponds to a second multi-head attention network; and
a fourth layer including a second feed forward network.

11. The device of claim 1 , further comprising a camera configured to generate the image data.

12. The device of claim 1 , further comprising a speaker recognition engine configured to generate speaker extraction data based on the input signal, the image data, or both, wherein the context data includes the speaker extraction data.

13. The device of claim 1 , further comprising an emotion recognition engine configured to generate emotion data based on the input signal, the image data, or both, wherein the context data includes the emotion data.

14. The device of claim 1 , further comprising a noise analysis engine configured to generate noise type data based on the input signal, the image data, or both, wherein the context data includes the noise type data.

15. The device of claim 1 , further comprising:
a microphone coupled to the one or more processors and configured to generate the input signal; and
a spectral analyzer configured to generate the input spectral data.

16. The device of claim 1 , wherein, to generate the context data, the one or more processors are configured to:
identify emotion data, speaker identification data, or noise type data as an in put embedding;
generate an embedding vector based on the input embedding; and
perform a linear projection on the embedding vector.

17. A method of speech enhancement, the method comprising:
processing image data to detect at least one of an emotion, a speaker identification, or a noise type;
generating context data that represents the at least one of the emotion, the speaker identification, or the noise type;
obtaining input spectral data based on an input signal that corresponds to the image data, the input signal representing sound that includes speech;
providing the input spectral data to a first encoder of a multi-encoder transformer to generate first encoded data;
providing the context data to at least a second encoder of the multi-encoder transformer to generate second encoded data;
providing the first encoded data and the second encoded data to a decoder of the multi-encoder transformer to generate output spectral data that represents a speech enhanced version of the input signal; and
performing speech synthesis on the output spectral data to generate an output waveform corresponding to an enhanced version of the speech.

18. The method of claim 17 , wherein providing the first encoded data and the second encoded data to the decoder includes:
providing the first encoded data and the second encoded data to a decoder attention network of the multi-encoder transformer to generate the output spectral data.

19. The method of claim 17 , further comprising obtaining the context data from one or more data sources, the one or more data sources including the input signal and the image data.

20. The method of claim 17 , further comprising:
obtaining the input signal from a microphone; and
processing the input signal at a spectral analyzer to generate the input spectral data.

21. The method of claim 17 , further comprising generating text based on the input signal, wherein the context data includes the text.

22. The method of claim 17 , further comprising generating speaker extraction data based on the input signal and the image data, wherein the context data includes the speaker extraction data.

23. The method of claim 17 , further comprising generating emotion data based on the input signal and the image data, wherein the context data includes the emotion data.

24. The method of claim 17 , further comprising generating noise type data based on the input signal and the image data, wherein the context data includes the noise type data.

25. The method of claim 17 , wherein generating the context data comprises:
identifying emotion data, speaker identification data, or noise type data as an input embedding;
generating an embedding vector based on the input embedding; and
performing a linear projection on the embedding vector.

26. A non-transitory computer-readable m

Assignees

• Qualcomm Inc

Inventors

• Byun Kyungguen
• Zhang Shuhua
• Kim Lae-Hoon
• Visser Erik
• Moon Sunkuk
• Montazeri Vahid

Classifications

• G06N3/0455

  Auto-encoder networks; Encoder-decoder networks · CPC title

• G10L21/02 Primary

  Speech enhancement, e.g. noise reduction or echo cancellation (reducing echo effects in line transmission systems H04B3/20; echo suppression in hands-free telephones H04M9/08) · CPC title

• G10L21/038

  using band spreading techniques · CPC title

• G10L13/08

  Text analysis or generation of parameters for speech synthesis out of text, e.g. grapheme to phoneme translation, prosody generation or stress or intonation determination · CPC title

• G10L25/18

  the extracted parameters being spectral information of each sub-band · CPC title