Systems and methods for speech transcription

Patent metadata

Field	Value
Publication number	US-10540957-B2
Application number	US-201514735002-A
Country	US
Kind code	B2
Filing date	Jun 9, 2015
Priority date	Dec 15, 2014
Publication date	Jan 21, 2020
Grant date	Jan 21, 2020

Abstract

Presented herein are embodiments of state-of-the-art speech recognition systems developed using end-to-end deep learning. In embodiments, the model architecture is significantly simpler than traditional speech systems, which rely on laboriously engineered processing pipelines; these traditional systems also tend to perform poorly when used in noisy environments. In contrast, embodiments of the system do not need hand-designed components to model background noise, reverberation, or speaker variation, but instead directly learn a function that is robust to such effects. A phoneme dictionary, nor even the concept of a “phoneme,” is needed. Embodiments include a well-optimized recurrent neural network (RNN) training system that can use multiple GPUs, as well as a set of novel data synthesis techniques that allows for a large amount of varied data for training to be efficiently obtained. Embodiments of the system can also handle challenging noisy environments better than widely used, state-of-the-art commercial speech systems.

First claim

What is claimed is:

1. A computer-implemented method for transcribing speech comprising:
receiving an input audio from a user;
normalizing the input audio to make a total power of the input audio consistent with a set of training samples used to train a trained neural network;
generating a jitter set of audio files from the normalized input audio by translating the normalized input audio by one or more time values;
for each audio file from the jitter set of audio files, which includes the normalized input audio:
generating a set of spectrogram frames for each audio file;
inputting the set of spectrogram frames into a trained neural network;
obtaining predicted character probabilities outputs from the trained neural network; and
decoding a transcription of the input audio using the predicted character probabilities outputs from the trained neural network constrained by a language model that interprets a string of characters from the predicted character probabilities outputs as a word or words.

2. The computer-implemented method of claim 1 wherein the step of generating a set of spectrogram frames for each audio file comprises:
generating spectrogram frames wherein a spectrogram frame comprises a set of linearly spaced log filter banks computed over windows of a first value of milliseconds strided by a second value of milliseconds.

3. The computer-implemented method of claim 1 wherein:
the step of inputting the set of spectrogram frames into a trained neural network comprises:
inputting the set of spectrogram frames into a set of trained neural networks; and

the step of obtaining predicted character probabilities outputs from the trained neural network comprises:
ensembling predicted character probabilities outputs from the set of trained neural networks to obtain the predicted character probabilities.

4. The computer-implemented method of claim 3 wherein the step of ensembling predicted character probabilities outputs from the set of trained neural networks to obtain the predicted character probabilities comprises:
addressing time shifts between trained neural networks by using one or more of the following comprising:
using trained neural networks that exhibit the same temporal shift;
shifting one or more of the outputs of the trained neural networks to align the outputs; and
shifting one or more of the inputs into one or more of the trained neural networks to have aligned outputs.

5. The computer-implemented method of claim 1 wherein the step of decoding a transcription of the input audio using the predicted character probabilities outputs from the trained neural network constrained by a language model that interprets the string of characters as words comprises:
given the predicted character probabilities outputs from the trained neural network, performing a search to find a sequence of characters that is most probable according to both the predicted character probabilities outputs and a trained N-gram language model output that interprets a string of characters from the predicted character probabilities outputs as a word or words.

6. The computer-implemented method of claim 1 wherein the trained neural network comprises a five-layer model comprising:
a first set of three layers that are non-recurrent;
a fourth layer that is a bi-directional recurrent network, which includes two sets of hidden units comprising a set with forward recurrence and a set with backward recurrence; and
a fifth layer that is a non-recurrent layer, which takes forward and backward units from the fourth layer as inputs and outputs the predicted character probabilities.

7. The computer-implemented method of claim 3 wherein the step of inputting the set of spectrogram frames into a trained neural network comprises:
inputting the set of spectrogram frames into the trained neural network in which at least one layer of the trained neural network operates on a context of spectrogram frames from the set of spectrogram frames.

8. A non-transitory computer-readable medium or media comprising one or more sequences of instructions which, when executed by one or more processors, causes the steps to be performed comprising:
receiving an input audio from a user;
generating a set of spectrogram frames from the input audio;
inputting the set of spectrogram frames into a set of trained neural networks;
obtaining predicted character probabilities outputs from the set of trained neural networks; and
decoding a transcription of the input audio using the predicted character probabilities outputs from the set of trained neural networks constrained by a language model that interprets a string of characters from the predicted character probabilities outputs as a word or words.

9. The non-transitory computer-readable medium or media of claim 7 wherein the step of generating a set of spectrogram frames comprises:
generating spectrogram frames wherein a spectrogram frame comprises a set of linearly spaced log filter banks computed over windows of a first value of milliseconds strided by a second value of milliseconds.

10. The non-transitory computer-readable medium or media of claim 8 wherein the step of obtaining predicted character probabilities outputs from the set of trained neural networks comprises:
ensembling predicted character probabilities outputs from the set of trained neural networks to obtain the predicted character probabilities.

11. The non-transitory computer-readable medium or media of claim 10 wherein the step of ensembling predicted character probabilities outputs from the set of trained neural networks to obtain the predicted character probabilities comprises:
addressing time shifts between trained neural networks by using one or more of the following comprising:
using trained neural networks that exhibit the same temporal shift;
shifting one or more of the outputs of the trained neural networks to align the outputs; and
shifting one or more of the inputs into one or more of the trained neural networks to have aligned outputs.

12. The non-transitory computer-readable medium or media of claim 8 wherein the step of generating a set of spectrogram frames from the input audio comprises:
generating a set of spectrogram frames from a normalized version of the input audio or from a normalized and jitter version of the input audio.

13. A computer-implemented method for transcribing speech comprising:
receiving an input audio from a user;
generating a set of spectrogram frames for the input audio;
inputting the set of spectrogram frames into a trained neural network;
obtaining predicted character probabilities outputs from the trained neural network; and
decoding a transcription of the input audio using the predicted character probabilities outputs from the trained neural network constrained by a language model that interprets a string of characters from the predicted character probabilities outputs as a word or words.

14. The computer-implemented method of claim 13 wherein the step of generating a set of spectrogram frames from the input audio comprises:
generating a set of spectrogram frames from a normalized version of the input audio or from a normalized and jitter version of the input audio.

15. The computer-implemented method of claim 14 wherein a normalized version of the input audio is obtained by performing the step comprising:
normalizing the input audio to make a total power of the input audio consistent with a set of training samples used to train the trained neural network.

16. The computer-implemented method of claim 13 wherein the step of in

Assignees

• Baidu Usa Llc

Inventors

• Hannun Awni
• Case Carl
• Casper Jared
• Catanzaro Bryan
• Diamos Gregory
• Elsen Erich
• Prenger Ryan
• Satheesh Sanjeev
• Sengupta Shubhabrata
• Coates Adam
• Ng Andrew Y

Classifications

• G06N3/045

  Combinations of networks · CPC title

• G06N3/044

  Recurrent networks, e.g. Hopfield networks · CPC title

• G10L15/26

  Speech to text systems (G10L15/08 takes precedence) · CPC title

• G06N3/084

  Backpropagation, e.g. using gradient descent · CPC title

• G10L15/16

  using artificial neural networks · CPC title