Multilingual wakeword detection

Patent metadata

Field	Value
Publication number	US-11069353-B1
Application number	US-201916404536-A
Country	US
Kind code	B1
Filing date	May 6, 2019
Priority date	May 6, 2019
Publication date	Jul 20, 2021
Grant date	Jul 20, 2021

Abstract

A system and method performs multilingual wakeword detection by determining a language corresponding to the wakeword. A first wakeword-detection component, which may execute using a digital-signal processor, determines that audio data includes a representation of the wakeword and determines a language corresponding to the wakeword. A second, more accurate wakeword-detection component may then process the audio data using the language to confirm that it includes the representation of the wakeword. The audio data may then be sent to a remote system for further processing.

First claim

What is claimed is:

1. A computer-implemented method comprising:
receiving, from at least one microphone, audio data;
determining, using a voice-activity detection (VAD) component, that the audio data includes a representation of speech;
determining, using a first wakeword-detection component executing on a digital-signal processor (DSP), that the audio data includes a first representation of a wakeword;
determining, using the first wakeword-detection component, language data indicating a first likelihood that the speech corresponds to a first spoken language;
after determining that the audio data includes the first representation of a wakeword, activating a second wakeword-detection component executing on a computer processor different from the DSP, at least a portion of the second wakeword-detection component corresponding to the first spoken language;
determining, using the second wakeword-detection component and the language data, that the audio data includes a second representation of the wakeword; and
sending, to a remote system, data representing the speech.

2. The computer-implemented method of claim 1 , wherein determining that the audio data includes the first representation of the wakeword comprises:
determining, using a shared recurrent neural network (RNN) and convolution neural network (CNN) encoder of the first wakeword-detection component, audio feature data corresponding to the audio data;
processing, using a wakeword classifier of the first wakeword-detection component, the audio feature data to determine a wakeword hypothesis; and
processing, using a language classifier of the first wakeword-detection component, the audio feature data to determine the language data.

3. The computer-implemented method of claim 1 , wherein determining the language data further comprises:
processing, using an acoustic model, the audio data to determine audio feature data;
processing, using an encoder, the audio feature data to determine encoded feature data; and
processing, using a language classifier, an encoded feature value to find a first value corresponding to the first likelihood.

4. The computer-implemented method of claim 1 , wherein determining that the audio data includes the second representation of the wakeword comprises:
determining, using an acoustic model of the second wakeword-detection component and the audio data, acoustic data corresponding to phonemes representing the first spoken language;
processing the acoustic data using a hidden Markov model corresponding to the first spoken language to determine a wakeword hypothesis; and
processing the wakeword hypothesis with a wakeword classifier corresponding to the first spoken language to determine that the audio data includes the second representation of the wakeword.

5. A computer-implemented method comprising:
receiving audio data representing speech;
determining, using a first wakeword-detection component, that the audio data includes a first representation of a wakeword;
determining, using the first wakeword-detection component, language data indicating a first likelihood that the audio data corresponds to a first spoken language;
determining, using a second wakeword-detection component and the language data, that the audio data includes a second representation of the wakeword; and
based at least in part on determining that the audio data includes the second representation of the wakeword, sending, to a remote system, data representing the speech.

6. The computer-implemented method of claim 5 , wherein determining that the audio data includes the first representation of the wakeword comprises:
determining audio feature data corresponding to the audio data, the audio feature data representing at least one audio feature of the audio data; and
processing, using a classifier, the audio feature data to determine a wakeword hypothesis.

7. The computer-implemented method of claim 5 , wherein determining the language data comprises:
determining audio feature data corresponding to the audio data, the audio feature data representing at least one audio feature of the audio data; and
processing, using a classifier, the audio feature data to determine the language data.

8. The computer-implemented method of claim 5 , wherein determining that the audio data includes the second representation of the wakeword comprises:
determining, using an acoustic model and the audio data, acoustic data corresponding to acoustic units representing the first spoken language;
processing the acoustic data using a hidden Markov model corresponding to the first spoken language to determine a wakeword hypothesis; and
processing the wakeword hypothesis with a classifier corresponding to the first spoken language to determine that the audio data includes the second representation of the wakeword.

9. The computer-implemented method of claim 5 , further comprising:
determining, using a voice-activity detection (VAD) component, that the audio data includes a representation of the speech;
based at least in part on determining that the audio data includes a representation of the speech, activating the first wakeword-detection component; and
based at least in part on determining that the audio data includes the first representation of the wakeword, activating the second wakeword-detection component.

10. The computer-implemented method of claim 5 , further comprising:
receiving, from the remote system, output data corresponding to a command represented in the speech; and
causing audio corresponding to the output data to be output.

11. The computer-implemented method of claim 10 , further comprising:
sending, to the remote system, the language data,
wherein the output data is based at least in part on the language data.

12. The computer-implemented method of claim 5 , further comprising:
receiving second audio data;
determining, using the first wakeword-detection component, that the second audio data includes a third representation of the wakeword;
determining, using the first wakeword-detection component, second language data indicating a second likelihood that the second audio data corresponds to a second spoken language;
determining, using an acoustic model and the second audio data, acoustic data corresponding to the second spoken language; and
processing the acoustic data using a hidden Markov model corresponding to the second spoken language to determine a wakeword hypothesis.

13. A system comprising:
at least one first processor;
at least one second processor; and
at least one memory including instructions that, when executed by the at least one first processor or the at least one second processor, cause the system to:
receive audio data representing speech;
determine, using a first wakeword-detection component, that the audio data includes a first representation of a wakeword;
determine, using the first wakeword-detection component, language data indicating a first likelihood that the audio data corresponds to a first spoken language;
determine, using a second wakeword-detection component and the language data, that the audio data includes a second representation of the wakeword; and
based at least in part on determining that the audio data includes the second representation of the wakeword, send, to a remote system, data representing the speech.

14. The system of claim 13 , wherein the at least one memory further includes instructions that, when executed by the at least one first processor or the at least one second processor, further cause the system to:
determine audio feature data corresponding to the audio data, the audio f

Assignees

• Amazon Tech Inc

Inventors

• Gao Yixin
• Sun Ming
• Krone Jason
• Vitaladevuni Shiv Naga Prasad
• Liu Yuzong

Classifications

• G10L2015/088

  Word spotting · CPC title

• G10L15/142

  Hidden Markov Models [HMMs] · CPC title

• G10L15/08 Primary

  Speech classification or search · CPC title

• G10L15/02

  Feature extraction for speech recognition; Selection of recognition unit · CPC title

• G10L15/005

  Language recognition · CPC title