User specified keyword spotting using long short term memory neural network feature extractor

Patent metadata

Field	Value
Publication number	US-9508340-B2
Application number	US-201414579603-A
Country	US
Kind code	B2
Filing date	Dec 22, 2014
Priority date	Dec 22, 2014
Publication date	Nov 29, 2016
Grant date	Nov 29, 2016

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for recognizing keywords using a long short term memory neural network. One of the methods includes receiving, by a device for each of multiple variable length enrollment audio signals, a respective plurality of enrollment feature vectors that represent features of the respective variable length enrollment audio signal, processing each of the plurality of enrollment feature vectors using a long short term memory (LSTM) neural network to generate a respective enrollment LSTM output vector for each enrollment feature vector, and generating, for the respective variable length enrollment audio signal, a template fixed length representation for use in determining whether another audio signal encodes another spoken utterance of the enrollment phrase by combining at most a quantity k of the enrollment LSTM output vectors for the enrollment audio signal.

First claim

What is claimed is:

1. A method comprising:
receiving, by a device for each of multiple variable length enrollment audio signals each encoding a respective spoken utterance of an enrollment phrase, a respective plurality of enrollment feature vectors that represent features of the respective variable length enrollment audio signal, wherein when the device determines that another audio signal encodes another spoken utterance of the enrollment phrase, the device performs a particular action assigned to the enrollment phrase; and
for each of the multiple variable length enrollment audio signals:
processing each of the plurality of enrollment feature vectors for the respective variable length enrollment audio signal using a long short term memory (LSTM) neural network to generate a respective enrollment LSTM output vector for each enrollment feature vector; and
generating, for the respective variable length enrollment audio signal, a template fixed length representation for use in determining whether the other audio signal encodes another spoken utterance of the enrollment phrase by combining at most a quantity k of the enrollment LSTM output vectors for the enrollment audio signal, wherein a predetermined length of each of the template fixed length representations is the same.

2. The method of claim 1 , comprising, for each of the multiple variable length enrollment audio signals:
determining whether at least the quantity k of enrollment feature vectors were generated for the respective enrollment audio signal; and
in response to determining that less than the quantity k of enrollment feature vectors were generated for the respective enrollment audio signal, adding leading zeros to a front of the respective template fixed length representation so that the respective template fixed length representation has the predetermined length.

3. The method of claim 2 , comprising determining an average number of enrollment frames in all of the enrollment audio signals wherein the quantity k comprises the average number of enrollment frames.

4. The method of claim 2 , wherein:
each of the enrollment output vectors has a predetermined size l that corresponds to a size of a last layer in the long short term memory neural network; and
adding leading zeros to the front of the respective template fixed length representation comprises adding leading zeros to the front of the respective template fixed length representation so that the respective template fixed length representation has a total of l times k values.

5. The method of claim 4 , wherein the last layer in the long short term memory neural network comprises a hidden layer during training of the long short term memory neural network.

6. The method of claim 1 , comprising, for at least one of the multiple variable length enrollment audio signals:
determining that more than the quantity k of enrollment LSTM output vectors were generated for the respective enrollment audio signal; and
in response, generating the template fixed length representation for the respective enrollment audio signal by combining the quantity k most recent enrollment LSTM output vectors.

7. A system comprising:
a computer and one or more storage devices storing instructions that are operable, when executed by the computer, to cause the computer to perform operations comprising:
receiving, by the computer for each of multiple variable length enrollment audio signals each encoding a respective spoken utterance of an enrollment phrase, a respective plurality of enrollment feature vectors that represent features of the respective variable length enrollment audio signal, wherein when the computer determines that another audio signal encodes another spoken utterance of the enrollment phrase, the computer performs a particular action assigned to the enrollment phrase; and
for each of the multiple variable length enrollment audio signals:
processing each of the plurality of enrollment feature vectors for the respective variable length enrollment audio signal using a long short term memory (LSTM) neural network to generate a respective enrollment LSTM output vector for each enrollment feature vector; and
generating, for the respective variable length enrollment audio signal, a template fixed length representation for use in determining whether the other audio signal encodes another spoken utterance of the enrollment phrase by combining at most a quantity k of the LSTM output vectors for the enrollment audio signal, wherein a predetermined length of each of the template fixed length representations is the same.

8. The system of claim 7 , the operations comprising:
receiving, for an audio signal encoding a spoken utterance of a phrase, a respective plurality of feature vectors each comprising values that represent features of the audio signal;
processing each of the feature vectors using the long short term memory neural network to generate a respective LSTM output vector for each of the feature vectors;
generating a fixed length representation for the audio signal by combining at most the quantity k of the LSTM output vectors; and
determining whether the phrase and the enrollment phrase are the same and the phrase was spoken by the same person using a comparison of the fixed length representation and the template fixed length representations.

9. The system of claim 8 , wherein determining whether the phrase and the enrollment phrase are the same using a comparison of the fixed length representation and all of the template fixed length representations comprises determining whether the phrase and the enrollment phrase are the same using a comparison of the fixed length representation and an average template fixed length representation created by averaging the values in each of the template fixed length representations to determine a corresponding value in the average template fixed length representation.

10. The system of claim 8 , wherein determining whether the phrase and the enrollment phrase are the same using a comparison of the fixed length representation and all of the template fixed length representations comprises determining a confidence score that represents a distance between the fixed length representation and the template fixed length representations.

11. The system of claim 10 , wherein determining the confidence score that represents the distance between the fixed length representation and the template fixed length representations comprises determining the distance between the fixed length representation and the template fixed length representations using a cosine distance function.

12. The system of claim 10 , the operations comprising determining that the confidence score satisfies a threshold value, wherein determining whether the phrase and the enrollment phrase are the same using a comparison of the representation and all of the template fixed length representations comprises determining that the phrase and the enrollment phrase are the same in response to determining that the confidence score satisfies the threshold value.

13. The system of claim 12 , the operations comprising:
receiving input indicating an action to perform in response to receipt of an audio signal encoding a spoken utterance of the enrollment phrase; and
performing the action in response to determining that the phrase and the enrollment phrase are the same.

14. The system of claim 13 , wherein:
receiving input indicating the action to perform in response to receipt of an audio signal encoding a spoken utterance of the enrollment phrase comprises receiving input indicating that when a particular device is asleep and receives an au

Assignees

• Google Inc

Inventors

• Parada San Martin Maria Carolina
• Sainath Tara N
• Chen Guoguo

Classifications

• G06N3/02

  Neural networks · CPC title

• G10L25/51

  for comparison or discrimination · CPC title

• G10L2015/088

  Word spotting · CPC title

• G10L2015/0631

  Creating reference templates; Clustering · CPC title

• G06F1/3203

  Power management, i.e. event-based initiation of a power-saving mode · CPC title