Shared speech processing network for multiple speech applications

Patent metadata

Field	Value
Publication number	US-11700484-B2
Application number	US-202217650595-A
Country	US
Kind code	B2
Filing date	Feb 10, 2022
Priority date	Apr 9, 2020
Publication date	Jul 11, 2023
Grant date	Jul 11, 2023

Abstract

A device to process speech includes a speech processing network that includes an input configured to receive audio data corresponding to audio captured by one or more microphones. The speech processing network also includes one or more network layers configured to process the audio data to generate a network output. The speech processing network includes an output configured to be coupled to multiple speech application modules to enable the network output to be provided as a common input to each of the multiple speech application modules. A first speech application module corresponds to a speaker verifier, and a second speech application module corresponds to a speech recognition network.

First claim

What is claimed is:

1. A device to process speech, comprising:
a speech processing network comprising:
an input configured to receive audio data corresponding to audio captured by one or more microphones;
one or more network layers configured to process the audio data to generate a network output; and
an output configured to be coupled to multiple speech application modules to enable the network output to be provided as a common input to each of the multiple speech application modules, wherein a first speech application module corresponds to a speaker verifier and a second speech application module corresponds to a speech recognition network.

2. The device of claim 1 , wherein the first speech application module is coupled to the output and configured to generate a speaker verification output based on the network output, and wherein the speaker verification output is indicative of whether the captured audio exhibits voice characteristics matching a particular speaker.

3. The device of claim 1 , wherein the second speech application module is coupled to the output and configured to generate a speech recognition output based on the network output, and wherein the speech recognition output includes text corresponding to the captured audio.

4. The device of claim 1 , further comprising a third speech application module coupled to the output, wherein the third speech application module corresponds to at least one of:
a voice activation detector;
a speaker recognizer; or
a speech enhancer.

5. The device of claim 1 , wherein the one or more network layers are trained based on at least a first performance metric associated with the first speech application module and a second performance metric associated with the second speech application module.

6. The device of claim 5 , wherein the one or more network layers are trained responsive to a combined performance metric that corresponds to a combination of at least the first performance metric and the second performance metric.

7. The device of claim 1 , wherein the speech processing network is implemented in an application-specific integrated circuit (ASIC), wherein at least one speech application module of the multiple speech application modules is external to the ASIC, and wherein the output of the speech processing network is coupled to a chip interface to enable the network output to be provided to the at least one speech application module that is external to the ASIC.

8. The device of claim 1 , wherein the input is configured to synchronously receive multiple frames of the audio data, each of the multiple frames received from a respective microphone of the one or more microphones, and wherein the speech processing network is configured to process the multiple frames to generate a single frame of the network output.

9. The device of claim 1 , wherein the speech processing network is included in a vehicle.

10. The device of claim 1 , wherein the speech processing network is implemented in an audio device, and wherein the audio device includes a wireless speaker and voice activated device with an integrated assistant application.

11. The device of claim 1 , further comprising:
an antenna; and
a transceiver coupled to the antenna and configured to receive the audio data via wireless transmission from the one or more microphones.

12. The device of claim 11 , wherein the speech processing network, the antenna, and the transceiver are integrated into a mobile device.

13. The device of claim 11 , wherein the speech processing network, the antenna, and the transceiver are integrated into a base station.

14. A method of speech processing, comprising:
receiving, at a speech processing network, audio data corresponding to audio captured by one or more microphones;
processing, at the speech processing network, the audio data using one or more network layers to generate a network output; and
providing the network output at an output of the speech processing network to enable the network output to be accessible as a common input to multiple speech application modules, wherein a first speech application module corresponds to a speaker verifier and a second speech application module corresponds to a speech recognition network.

15. The method of claim 14 , further comprising generating, at the first speech application module, a speaker verification output based on the network output and indicative of whether the captured audio exhibits voice characteristics matching a particular speaker.

16. The method of claim 14 , further comprising generating, at the second speech application module, a speech recognition output based on the network output and including text corresponding to the captured audio.

17. The method of claim 14 , further comprising providing the network output to a third speech application module to generate at least one of:
a voice activation output;
a speaker recognition output; or
an enhanced speech output.

18. The method of claim 14 , further comprising training the speech processing network based on at least a first performance metric associated with the first speech application module and a second performance metric associated with the second speech application module.

19. The method of claim 18 , wherein the speech processing network is trained responsive to a combined performance metric that corresponds to a combination of at least the first performance metric and the second performance metric.

20. The method of claim 14 , wherein the network output is provided to a chip interface of an application-specific integrated circuit (ASIC) to enable the network output to be provided to at least one speech application module of the multiple speech application modules that is external to the ASIC.

21. The method of claim 14 , wherein the audio data is received as multiple frames that are synchronously received from respective microphones of the one or more microphones, and wherein the speech processing network processes the multiple frames to generate a single frame of the network output.

22. A non-transitory computer-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to:
receive, at a speech processing network, audio data corresponding to audio captured by one or more microphones;
process, at the speech processing network, the audio data using one or more network layers to generate a network output; and
provide the network output at an output of the speech processing network to enable the network output to be accessible as a common input to multiple speech application modules, wherein a first speech application module corresponds to a speaker verifier and a second speech application module corresponds to a speech recognition network.

23. The non-transitory computer-readable medium of claim 22 , wherein execution of the instructions further causes the one or more processors to:
provide the network output to the first speech application module to generate a speaker verification output based on the network output; and
provide the network output to the second speech application module to generate a speech recognition output based on the network output.

24. The non-transitory computer-readable medium of claim 23 , wherein execution of the instructions further causes the one or more processors to provide the network output to a third speech application module to generate at least one of:
a voice activation output;

Assignees

• Qualcomm Inc

Inventors

• Kim Lae-Hoon
• Moon Sunkuk
• Visser Erik
• Kulkarni Prajakt

Classifications

• G10L21/02

  Speech enhancement, e.g. noise reduction or echo cancellation (reducing echo effects in line transmission systems H04B3/20; echo suppression in hands-free telephones H04M9/08) · CPC title

• H04R3/005 Primary

  for combining the signals of two or more microphones (specially adapted for hearing aids H04R25/407) · CPC title

• G06V10/82

  using neural networks · CPC title

• G06V20/20

  in augmented reality scenes · CPC title

• H04R2499/13

  Acoustic transducers and sound field adaptation in vehicles · CPC title