Context-based speech enhancement

Patent metadata

Field	Value
Publication number	US-11715480-B2
Application number	US-202117209621-A
Country	US
Kind code	B2
Filing date	Mar 23, 2021
Priority date	Mar 23, 2021
Publication date	Aug 1, 2023
Grant date	Aug 1, 2023

Abstract

A device to perform speech enhancement includes one or more processors configured to obtain input spectral data based on an input signal. The input signal represents sound that includes speech. The one or more processors are also configured to process, using a multi-encoder transformer, the input spectral data and context data to generate output spectral data that represents a speech enhanced version of the input signal.

First claim

What is claimed is:

1. A device to perform speech enhancement, the device comprising:
one or more processors configured to:
obtain input spectral data based on an input signal, the input signal representing sound that includes speech; and
process, using a multi-encoder transformer, the input spectral data and context data to generate output spectral data that represents a speech enhanced version of the input signal,
wherein the multi-encoder transformer includes:
a multi-encoder that includes:
a first encoder that includes a first attention network;
at least a second encoder that includes a second attention network; and

a decoder that includes a decoder attention network,
wherein the decoder further comprises:
a masked multi-head attention network coupled to an input of the decoder attention network; and
a decoder feed forward network coupled to an output of the decoder attention network.

2. The device of claim 1 , wherein the one or more processors are configured to:
provide the input spectral data to the first encoder to generate first encoded data;
obtain the context data based on one or more data sources;
provide the context data to at least the second encoder to generate second encoded data; and
provide, to the decoder attention network, the first encoded data and the second encoded data to generate output spectral data that corresponds to a speech enhanced version of the input spectral data.

3. The device of claim 2 , wherein the one or more data sources includes at least one of the input signal or image data.

4. The device of claim 3 , further comprising a camera configured to generate the image data.

5. The device of claim 2 , wherein the decoder attention network comprises:
a first multi-head attention network configured to process the first encoded data;
a second multi-head attention network configured to process the second encoded data; and
a combiner configured to combine outputs of the first multi-head attention network and the second multi-head attention network.

6. The device of claim 1 , the first encoder including a Mel filter bank configured to filter the input spectral data.

7. The device of claim 1 , further comprising an automatic speech recognition engine configured to generate text based on the input signal, wherein the context data includes the text.

8. The device of claim 7 , wherein the second encoder includes a grapheme-to-phoneme convertor configured to process the text.

9. The device of claim 1 , wherein:
the first encoder comprises:
a first layer including the first attention network, wherein the first attention network corresponds to a first multi-head attention network; and
a second layer including a first feed forward network, and

the second encoder comprises:
a first layer including the second attention network, wherein the second attention network corresponds to a second multi-head attention network; and
a second layer including a second feed forward network.

10. The device of claim 1 , further comprising a speaker recognition engine configured to generate speaker extraction data based on the input signal, and wherein the context data includes the speaker extraction data.

11. The device of claim 1 , further comprising an emotion recognition engine configured to generate emotion data based on the input signal, and wherein the context data includes the emotion data.

12. The device of claim 1 , further comprising a noise analysis engine configured to generate noise type data based on the input signal, and wherein the context data includes the noise type data.

13. The device of claim 1 , further comprising:
a microphone coupled to the one or more processors and configured to generate the input signal; and
a spectral analyzer configured to generate the input spectral data.

14. The device of claim 1 , further comprising a waveform generator configured to process the output spectral data to generate an output waveform corresponding to an enhanced version of the speech.

15. A method of speech enhancement, the method comprising:
obtaining input spectral data based on an input signal, the input signal representing sound that includes speech; and
processing, using a multi-encoder transformer, the input spectral data and context data to generate output spectral data that represents a speech enhanced version of the input signal,
wherein processing, using the multi-encoder transformer, includes:
encoding, using a multi-encoder, wherein the encoding includes:
a first encoding by a first encoder using a first attention network;
at least a second encoding by a second encoder using a second attention network; and

decoding, using a decoder attention network, wherein the decoding includes:
providing an output of a masked multi-head attention network to an input of the decoder attention network; and
providing an output of the decoder attention network to a decoder feed forward network.

16. The method of claim 15 , wherein processing the input spectral data includes:
providing the input spectral data to the first encoder of the multi-encoder transformer to generate first encoded data;
providing the context data to at least the second encoder of the multi-encoder transformer to generate second encoded data; and
providing the first encoded data and the second encoded data to the decoder attention network of the multi-encoder transformer to generate output spectral data that corresponds to a speech enhanced version of the input spectral data.

17. The method of claim 15 , further comprising obtaining the context data from one or more data sources, the one or more data sources including at least one of the input signal or image data.

18. The method of claim 15 , further comprising:
obtaining the input signal from a microphone; and
processing the input signal at a spectral analyzer to generate the input spectral data.

19. The method of claim 15 , further comprising generating text based on the input signal, wherein the context data includes the text.

20. The method of claim 15 , further comprising generating speaker extraction data based on the input signal, and wherein the context data includes the speaker extraction data.

21. The method of claim 15 , further comprising generating emotion data based on the input signal, and wherein the context data includes the emotion data.

22. The method of claim 15 , further comprising generating noise type data based on the input signal, and wherein the context data includes the noise type data.

23. The method of claim 15 , further comprising processing the output spectral data to generate an output waveform corresponding to an enhanced version of the speech.

24. A non-transitory computer-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to:
obtain input spectral data based on an input signal, the input signal representing sound that includes speech; and
process, using a multi-encoder transformer, the input spectral data and context data to generate output spectral data that represents a speech enhanced version of the input signal,
wherein processing, using the multi-encoder transformer includes:
encoding, using a multi-encoder, wherein the encoding includes:
a first encoding by a first encoder using a first attention network;
at least a second encoding by a second encoder using a second attention network; and

decoding, using a decoder attentio

Assignees

• Qualcomm Inc

Inventors

• Byun Kyungguen
• Zhang Shuhua
• Kim Lae-Hoon
• Visser Erik
• Moon Sunkuk
• Montazeri Vahid

Classifications

• G06N3/0455

  Auto-encoder networks; Encoder-decoder networks · CPC title

• G10L25/30

  using neural networks · CPC title

• G10L21/02 Primary

  Speech enhancement, e.g. noise reduction or echo cancellation (reducing echo effects in line transmission systems H04B3/20; echo suppression in hands-free telephones H04M9/08) · CPC title

• G10L25/63

  for estimating an emotional state · CPC title

• G10L25/18

  the extracted parameters being spectral information of each sub-band · CPC title