Systems and methods for principled bias reduction in production speech models

Patent metadata

Field	Value
Publication number	US-10657955-B2
Application number	US-201815884239-A
Country	US
Kind code	B2
Filing date	Jan 30, 2018
Priority date	Feb 24, 2017
Publication date	May 19, 2020
Grant date	May 19, 2020

Abstract

Described herein are systems and methods to identify and address sources of bias in an end-to-end speech model. In one or more embodiments, the end-to-end model may be a recurrent neural network with two 2D-convolutional input layers, followed by multiple bidirectional recurrent layers and one fully connected layer before a softmax layer. In one or more embodiments, the network is trained end-to-end using the CTC loss function to directly predict sequences of characters from log spectrograms of audio. With optimized recurrent layers and training together with alignment information, some unwanted bias induced by using purely forward only recurrences may be removed in a deployed model.

First claim

What is claimed is:

1. A computer-implemented method for automatic speech recognition comprising:
receiving an input audio comprising an utterance;
generating a set of spectrogram frames for the utterance;
dividing a representation of the utterance related to the set of spectrogram frames into a plurality of chunks that overlap;
processing the representation of the utterance in forward recurrence in a recurrent neural network (RNN);
processing at least some of the plurality of chunks in backward recurrence in the RNN; and
using hidden states of the backward recurrences obtained from at least some of the chunks to calculate a final output of the RNN.

2. The computer-implemented method of claim 1 wherein the plurality of chunks are divided uniformly with each chunk corresponding to a fixed context size.

3. The computer-implemented method of claim 1 wherein backward recurrence starts processing with a first chunk of the plurality of chunks and moves ahead by a chunk step size that is less than a chunk size.

4. The computer-implemented method of claim 1 wherein the hidden states of the backward recurrences are reset between each chunk.

5. The computer-implemented method of claim 1 wherein the RNN comprises one or more gated recurrent unit (GRU) layers.

6. The computer-implemented method of claim 5 wherein the one or more gated recurrent layers comprise at least one Latency Constrained Bidirectional GRU (LC-BGRU) layer to process the plurality of chunks in backward recurrence.

7. The computer-implemented method of claim 1 wherein the set of spectrogram frames for the utterance are used as an input to a per-channel energy normalization (PCEN) process.

8. The computer-implemented method of claim 1 further comprising:
training the RNN with a cross-entropy (CE) loss for one or more epochs; and
training the RNN by implementing a Gram Connectionist Temporal Classification (Gram-CTC) loss function in a CTC cost layer using the final output of the RNN.

9. The computer-implemented method of claim 1 wherein the Gram-CTC loss function automatically selects useful grams from a pre-specified set and decomposes an input utterance with the selected grams.

10. The computer-implemented method of claim 9 wherein the pre-specified set comprises uni-grams and selected bi-grams and tri-grams.

11. A computer-implemented method for training a neural network (NN) model for speech transcription, the method comprising:
inputting an utterance from a training set into the NN model, the NN model comprising one or more convolutional layers and a bidirectional recurrent layer;
generating a representation of the utterance that uses a trainable per-channel energy normalization (PCEN) process;
using bidirectional recurrent layer, processing the representation of the utterance in a forward recurrence and processing at least some of a plurality of chunks of the representation in a backward recurrence fashion to obtain an output of the NN;
obtaining a loss for the NN using the output of the NN and a Connectionist Temporal Classification (CTC) loss function; and
using the loss to update at least a portion of the NN.

12. The computer-implemented method of claim 11 wherein the backward recurrence fashion starts processing with a first chunk of the plurality of chunks and moves ahead by a chunk step size that is less than a chunk size.

13. The computer-implemented method of claim 12 wherein the bidirectional recurrent layer comprises a Latency Constrained Bidirectional GRU (LC-BGRU) layer.

14. The computer-implemented method of claim 11 wherein the RNN NN is trained simultaneously using a weighted combination of a CTC loss and a cross-entropy (CE) loss.

15. The computer-implemented method of claim 14 wherein the CTC loss is a Gram-CTC loss, and the Gram-CTC loss and the CE loss have equal weighting factors.

16. A computer-implemented method for speech transcription with a recurrent neural network (RNN), the method comprising:
receiving a set of spectrogram frames correspond to an utterance,
dividing a representation of the utterance into a plurality of chunks that overlap, each chunk corresponding to a fixed context size;
processing the representation of the utterance in forward recurrence in one or more gated recurrent unit (GRU) layers within the RNN;
processing at least some of the plurality of chunks in a backwards recurrent fashion in the one or more GRU layers to obtain a plurality of hidden states; and
using the obtained hidden states to calculate a final output of the RNN.

17. The computer-implemented method of claim 16 wherein at least some of the chunks of the plurality of chunks are divided uniformly.

18. The computer-implemented method of claim 16 wherein each of the at least some of the plurality of chunks is processed independently in backward recurrence with hidden states reset between each chunk.

19. The computer-implemented method of claim 16 wherein the RNN is deployed for inference in a streaming setting.

20. The computer-implemented method of claim 16 wherein the RNN was trained using a weighted combination of a Gram Connectionist Temporal Classification (CTC) loss and a cross-entropy (CE) loss.

Assignees

• Baidu Usa Llc

Inventors

• Battenberg Eric
• Child Rewon
• Coates Adam
• Fougner Christopher
• Gaur Yashesh
• Huang Jiaji
• Jun Heewoo
• Kannan Ajay
• Kliegl Markus
• Kumar Atul
• Liu Hairong
• Rao Vinay
• Satheesh Sanjeev
• Seetapun David
• Sriram Anuroop
• Zhu Zhenyao

Classifications

• G10L15/04

  Segmentation; Word boundary detection · CPC title

• G10L15/06

  Creation of reference templates; Training of speech recognition systems, e.g. adaptation to the characteristics of the speaker's voice (G10L15/14 takes precedence) · CPC title

• G10L15/16 Primary

  using artificial neural networks · CPC title

• G10L15/063

  Training · CPC title

• G10L25/18

  the extracted parameters being spectral information of each sub-band · CPC title