Shared speech processing network for multiple speech applications

Patent metadata

Field	Value
Publication number	US-11276415-B2
Application number	US-202016844836-A
Country	US
Kind code	B2
Filing date	Apr 9, 2020
Priority date	Apr 9, 2020
Publication date	Mar 15, 2022
Grant date	Mar 15, 2022

Abstract

A device to process speech includes a speech processing network that includes an input configured to receive audio data corresponding to audio captured by one or more microphones. The speech processing network also includes one or more network layers configured to process the audio data to generate an output representation of the audio data. The speech processing network includes an output configured to be coupled to multiple speech application modules to enable the output representation to be provided as a common input to each of the multiple speech application modules.

First claim

What is claimed is:

1. A device to process speech, the device comprising:
a speech processing network comprising:
an input configured to receive audio data corresponding to audio captured by one or more microphones;
one or more network layers configured to process the audio data to generate an embedding that represents the audio data; and
an output configured to be coupled to multiple speech application modules to enable the embedding to be provided as a common input to each of the multiple speech application modules.

2. The device of claim 1 , further comprising:
a first speech application module coupled to the output and configured to generate a first application output based on the embedding; and
a second speech application module coupled to the output and configured to generate a second application output based on the embedding, the second application output different than the first application output.

3. The device of claim 2 , wherein the first speech application module corresponds to at least one of:
a speech recognition network;
a voice activation detector;
a speaker verifier;
a speaker recognizer; or
a speech enhancer.

4. The device of claim 1 , wherein the one or more network layers are trained based on at least a first performance metric associated with a first speech application module and a second performance metric associated with a second speech application module.

5. The device of claim 4 , wherein the one or more network layers are trained responsive to a combined performance metric that corresponds to a combination of at least the first performance metric and the second performance metric.

6. The device of claim 4 , wherein the first speech application module includes at least one of a speech enhancement module, a speech recognition module, or speaker recognition module.

7. The device of claim 1 , wherein the speech processing network is implemented in an application-specific integrated circuit (ASIC), wherein at least one speech application module of the multiple speech application modules is external to the ASIC, and wherein the output of the speech processing network is coupled to a chip interface to enable the embedding to be provided to the at least one speech application module that is external to the ASIC.

8. The device of claim 1 , wherein the input is configured to synchronously receive multiple frames of the audio data, each of the multiple frames received from a respective microphone of the one or more microphones, and wherein the speech processing network is configured to process the multiple frames to generate a single frame of the embedding.

9. The device of claim 1 , wherein the speech processing network is included in a vehicle.

10. The device of claim 1 , wherein the speech processing network is implemented in an audio device, and wherein the audio device includes a wireless speaker and voice activated device with an integrated assistant application.

11. The device of claim 1 , further comprising:
an antenna; and
a transceiver coupled to the antenna and configured to receive the audio data via wireless transmission from the one or more microphones.

12. The device of claim 11 , wherein the speech processing network, the antenna, and the transceiver are integrated into a mobile device.

13. The device of claim 11 , wherein the speech processing network, the antenna, and the transceiver are integrated into a base station.

14. A method of speech processing, comprising:
receiving, at a speech processing network, audio data corresponding to audio captured by one or more microphones;
processing, at the speech processing network, the audio data using one or more network layers to generate an embedding that represents the audio data; and
providing the embedding at an output of the speech processing network to enable the embedding to be accessible as a common input to multiple speech application modules.

15. The method of claim 14 , further comprising:
providing the embedding to a first speech application module to generate a first application output based on the embedding; and
providing the embedding to a second speech application module to generate a second application output based on the embedding, the second application output different than the first application output.

16. The method of claim 15 , wherein the first application output corresponds to at least one of:
a speech recognition output;
a voice activation output;
a speaker verification output;
a speaker recognition output; or
an enhanced speech output.

17. The method of claim 14 , further comprising training the speech processing network based on at least a first performance metric associated with a first speech application module and a second performance metric associated with a second speech application module.

18. The method of claim 17 , wherein the speech processing network is trained responsive to a combined performance metric that corresponds to a combination of at least the first performance metric and the second performance metric.

19. The method of claim 17 , wherein the first speech application module includes at least one of a speech enhancement module, a speech recognition module, or a speaker recognition module.

20. The method of claim 14 , wherein the embedding is provided to a chip interface of an application-specific integrated circuit (ASIC) to enable the embedding to be provided to at least one speech application module of the multiple speech application modules that is external to the ASIC.

21. The method of claim 14 , wherein the audio data is received as multiple frames that are synchronously received from respective microphones of the one or more microphones, and wherein the speech processing network processes the multiple frames to generate a single frame of the embedding.

22. A non-transitory computer-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to:
receive, at a speech processing network, audio data corresponding to audio captured by one or more microphones;
process, at the speech processing network, the audio data using one or more network layers to generate an embedding that represents the audio data; and
provide the embedding at an output of the speech processing network to enable the embedding to be accessible as a common input to multiple speech application modules.

23. The non-transitory computer-readable medium of claim 22 , wherein execution of the instructions further causes the one or more processors to:
provide the embedding to a first speech application module to generate a first application output based on the embedding; and
provide the embedding to a second speech application module to generate a second application output based on the embedding, the second application output different than the first application output.

24. The non-transitory computer-readable medium of claim 23 , wherein the first application output corresponds to at least one of:
a speech recognition output;
a voice activation output;
a speaker verification output;
a speaker recognition output; or
an enhanced speech output.

25. The non-transitory computer-readable medium of claim 22 , wherein execution of the instructions further causes the one or more processors to train the speech processing network based on at least a first performance metric associated with a first speech application module and a se

Assignees

• Qualcomm Inc

Inventors

• Kim Lae-Hoon
• Moon Sunkuk
• Visser Erik
• Kulkarni Prajakt

Classifications

• H04R3/005 Primary

  for combining the signals of two or more microphones (specially adapted for hearing aids H04R25/407) · CPC title

• G10L21/02 Primary

  Speech enhancement, e.g. noise reduction or echo cancellation (reducing echo effects in line transmission systems H04B3/20; echo suppression in hands-free telephones H04M9/08) · CPC title

• G06V20/20

  in augmented reality scenes · CPC title

• G06V10/82

  using neural networks · CPC title

• G06N3/044

  Recurrent networks, e.g. Hopfield networks · CPC title