Dynamic combination of acoustic model states

Patent metadata

Field	Value
Publication number	US-12014728-B2
Application number	US-201916363705-A
Country	US
Kind code	B2
Filing date	Mar 25, 2019
Priority date	Mar 25, 2019
Publication date	Jun 18, 2024
Grant date	Jun 18, 2024

Abstract

A computer implemented method classifies an input corresponding to multiple different kinds of input. The method includes obtaining a set of features from the input, providing the set of features to multiple different models to generate state predictions, generating a set of state-dependent predicted weights, and combining the state predictions from the multiple models, based on the state-dependent predicted weights for classification of the set of features.

First claim

The invention claimed is:

1. A computer-implemented method for classification of input corresponding to multiple different kinds of input, the method comprising:
obtaining a set of features from the input, wherein the input comprises speech and the set of features comprise digital representations of speech;
providing the set of features to multiple different acoustic models that have been trained on different kinds of input, or have different structures or modeling technologies, to generate state predictions, wherein the acoustic models comprise hidden layer outputs;
generating a set of state-dependent predicted weights from the hidden layer outputs;
combining the state predictions from the multiple acoustic models, based on the state-dependent predicted weights, for classification of the set of features;
and providing the combined state predictions to a speech decoder to classify the input as one or more spoken words;
wherein the set of state-dependent predicted weights are dynamic weights;
and wherein the hidden layer outputs are concatenated hidden layer outputs that are generated by providing the hidden layer outputs from the different acoustic models to a concatenation function.

2. The method of claim 1 , wherein the multiple acoustic models are speech recognition models independently trained on the same or different kinds of speech, and wherein the different kinds of speech include two or more of speech in a noisy environment, native speech, non-native speech, child speech, whispered speech, natural conversation speech and distant speech.

3. The method of claim 1 wherein the multiple models comprise deep long-term short memory (LSTM) acoustic models.

4. The method of claim 1 wherein generating the set of state dependent predicted weights is performed by a deep learning network trained on a training set of state labeled features.

5. The method of claim 1 wherein the set of predicted weights are time dependent static weights.

6. The method of claim 1 , wherein the set of predicted dynamic weights are provided by one of the acoustic models comprising Bidirectional long-term short memory (LSTM), generic Recurrent Neural Networks (RNN), or Convolution Neural Networks (CNN).

7. A machine-readable storage device having instructions for execution by a processor of a machine to cause the processor to perform operations to perform a method of classifying different kinds of input, the operations comprising:
obtaining a set of features from the input, wherein the input comprises speech and the set of features comprise digital representations of speech;
providing the set of features to multiple different acoustic models that have been trained on different kinds of input, or have different structures or modeling technologies, to generate state predictions, wherein the acoustic models comprise hidden layer outputs;
generating a set of state-dependent predicted weights from the hidden layer outputs;
combining the state predictions from the multiple acoustic models, based on the state-dependent predicted weights for classification of the set of features;
and providing the combined state predictions to a speech decoder to classify the input as one or more spoken words;
wherein the set of state-dependent predicted weights are dynamic weights;
and wherein the hidden layer outputs are concatenated hidden layer outputs that are generated by providing the hidden layer outputs from the different acoustic models to a concatenation function.

8. The device of claim 7 , wherein the multiple acoustic models are speech recognition models independently trained on the same or different kinds of speech, and wherein the different kinds of speech include two or more of speech in a noisy environment, native speech, non-native speech, child speech, whispered speech, natural conversation speech; and distant speech.

9. A device comprising:
a processor; and
a memory device coupled to the processor and having a program stored thereon for execution by the processor to perform operations comprising:
obtaining a set of features from the input, wherein the input comprises speech and the set of features comprise digital representations of speech;
providing the set of features to multiple different acoustic models that have been trained on different kinds of input, or have different structures or modeling technologies, to generate state predictions, wherein the acoustic models comprise hidden layer outputs;
generating a set of state-dependent predicted weights from the hidden layer outputs;
combining the state predictions from the multiple acoustic models, based on the state-dependent predicted weights for classification of the set of features;
and providing the combined state predictions to a speech decoder to classify the input as one or more spoken words;
wherein the set of state-dependent predicted weight are dynamic weights;
wherein the hidden layer outputs ae concatenated hidden layer outputs that are generated by providing the hidden layer outputs from the different acoustic models to a concatenation function.

10. The device of claim 9 , wherein the multiple acoustic models are speech recognition models independently trained on the same or different kinds of speech, and wherein the different kinds of speech include two or more of speech in a noisy environment, native speech, non-native speech, child speech, whispered speech, natural conversation speech and distant speech.

11. The device of claim 9 wherein generating the set of state dependent predicted weights is performed by a trained deep learning network trained on a training set of state labeled features.

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Kumar Kshitiz
• Gong Yifan

Classifications

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/0442

  characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU] · CPC title

• G06N3/09

  Supervised learning · CPC title

• G10L15/02

  Feature extraction for speech recognition; Selection of recognition unit · CPC title

• G10L25/30

  using neural networks · CPC title