Fast emit low-latency streaming ASR with sequence-level emission regularization utilizing forward and backward probabilities between nodes of an alignment lattice

Patent metadata

Field	Value
Publication number	US-12094453-B2
Application number	US-202117447285-A
Country	US
Kind code	B2
Filing date	Sep 9, 2021
Priority date	Oct 20, 2020
Publication date	Sep 17, 2024
Grant date	Sep 17, 2024

Abstract

A computer-implemented method of training a streaming speech recognition model that includes receiving, as input to the streaming speech recognition model, a sequence of acoustic frames. The streaming speech recognition model is configured to learn an alignment probability between the sequence of acoustic frames and an output sequence of vocabulary tokens. The vocabulary tokens include a plurality of label tokens and a blank token. At each output step, the method includes determining a first probability of emitting one of the label tokens and determining a second probability of emitting the blank token. The method also includes generating the alignment probability at a sequence level based on the first probability and the second probability. The method also includes applying a tuning parameter to the alignment probability at the sequence level to maximize the first probability of emitting one of the label tokens.

First claim

What is claimed is:

1. A computer-implemented method when executed on data processing hardware causes the data processing hardware to perform operations for training a streaming speech recognition model, the operations comprising:
receiving, as input to the streaming speech recognition model, a sequence of acoustic frames, the streaming speech recognition model configured to learn an alignment probability between the sequence of acoustic frames and an output sequence of vocabulary tokens, the vocabulary tokens comprising a plurality of label tokens and a blank token;
generating an alignment lattice comprising a plurality of nodes, the alignment lattice defined as a matrix with T columns of nodes and U rows of nodes, each column of the T columns corresponding to a corresponding step of the plurality of output steps, each row of the U rows corresponding to a label that textually represents the sequence of acoustic frames;
at each node location in the matrix of the alignment lattice:
determining a forward probability for predicting a subsequent node adjacent to the respective node; and
determining, from the subsequent node adjacent to the respective node, a backward probability of including the respective subsequent node in an output sequence of vocabulary tokens;

at each step of a plurality of output steps:
determining a first probability of emitting one of the label tokens; and
determining a second probability of emitting the blank token, wherein the forward probability comprises the first probability and the second probability;

generating the alignment probability at a sequence level based on the first probability of emitting one of the label tokens and the second probability of emitting the blank token at each output step; and
applying a tuning parameter to the alignment probability at the sequence level to maximize the first probability of emitting one of the label tokens, the tuning parameter applied to the alignment probability independent of any speech-word alignment information.

2. The computer-implemented method of claim 1 , wherein the first probability of emitting one of the label tokens at a respective step corresponds to a probability of emitting one of the label tokens after previously emitting a respective label token.

3. The computer-implemented method of claim 1 , wherein the second probability of emitting the blank token at a respective step corresponds to a probability of emitting the blank token after emitting one of the blank token or a label token at a step immediately preceding the respective step.

4. The computer-implemented method of claim 1 , wherein the first probability and the second probability define a forward variable of a forward-backward propagation algorithm.

5. The computer-implemented method of claim 1 , wherein generating the alignment probability at the sequence level comprises aggregating the forward probability and the backward probability for all nodes at each respective step of the alignment lattice.

6. The computer-implemented method of claim 1 , wherein applying the tuning parameter to the alignment probability at the sequence level balances a loss at the streaming speech recognition model and a regularization loss when training the streaming speech recognition model.

7. The computer-implemented method of claim 1 , wherein emission of the blank token at one of the output steps is not penalized.

8. The computer-implemented method of claim 1 , wherein the streaming speech recognition model comprises at least one of:
a recurrent neural-transducer (RNN-T) model;
a Transformer-Transducer model;
a Convolutional Network-Transducer (ConvNet-Transducer) model; or
a Conformer-Transducer model.

9. The computer-implemented method of claim 1 , wherein the streaming speech recognition model comprises a recurrent neural-transducer (RNN-T) model.

10. The computer-implemented method of claim 1 , wherein the streaming speech recognition model comprises a Conformer-Transducer model.

11. The computer-implemented method of claim 1 , wherein after training the streaming speech recognition model, the trained streaming speech recognition model executes on a user device to transcribe speech in a streaming fashion.

12. The computer-implemented method of claim 1 , wherein, after training the streaming speech recognition model, the trained streaming speech recognition model executes on a server.

13. A system of training a streaming speech recognition model, the system comprising:
data processing hardware; and
memory hardware in communication with the data processing hardware, the memory hardware storing instructions that when executed by the data processing hardware cause the data processing hardware to perform operations comprising:
receiving, as input to the speech recognition model, a sequence of acoustic frames, the streaming speech recognition model configured to learn an alignment probability between the sequence of acoustic frames and an output sequence of vocabulary tokens, the vocabulary tokens comprising a plurality of label tokens and a blank token;
generating an alignment lattice comprising a plurality of nodes, the alignment lattice defined as a matrix with T columns of nodes and U rows of nodes, each column of the T columns corresponding to a corresponding step of the plurality of output steps, each row of the U rows corresponding to a label that textually represents the sequence of acoustic frames;
at each node location in the matrix of the alignment lattice:
determining a forward probability for predicting a subsequent node adjacent to the respective node; and
determining, from the subsequent node adjacent to the respective node, a backward probability of including the respective subsequent node in an output sequence of vocabulary tokens;

at each step of a plurality of output steps:
determining a first probability of emitting one of the label tokens; and
determining a second probability of emitting the blank token, wherein the forward probability comprises the first probability and the second probability;

generating the alignment probability at a sequence level based on the first probability of emitting one of the label tokens and the second probability of emitting the blank token at each output step; and
applying a tuning parameter to the alignment probability at the sequence level to maximize the first probability of emitting one of the label tokens, the tuning parameter applied to the alignment probability independent of any speech-word alignment information.

14. The system of claim 13 , wherein the first probability of emitting one of the label tokens at a respective step corresponds to a probability of emitting one of the label tokens after previously emitting a respective label token.

15. The system of claim 13 , wherein the second probability of emitting the blank token at a respective step corresponds to a probability of emitting the blank label after emitting one of the blank label or a label token at a step immediately preceding the respective step.

16. The system of claim 13 , wherein the first probability and the second probability define a forward variable of a forward-backward propagation algorithm.

17. The system of claim 13 , wherein generating the alignment probability at the sequence level comprises aggregating the forward probability and the backward probability for all nodes at each respective step of the alignment lattice.

18. The system of claim 13 , wherein applying the tuning parameter to the alignment probability at the sequence level balances a loss at the streaming spee

Assignees

• Google Llc

Inventors

• Yu Jiahui
• Chiu Chung-Cheng
• Li Bo
• Chang Shuo-Yiin
• Sainath Tara
• Han Wei
• Gulati Anmol
• He Yanzhang
• Narayanan Arun
• Wu Yonghui
• Pang Ruoming

Classifications

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06N3/0442

  characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU] · CPC title

• G10L15/187

  Phonemic context, e.g. pronunciation rules, phonotactical constraints or phoneme n-grams · CPC title

• G10L15/30

  Distributed recognition, e.g. in client-server systems, for mobile phones or network applications · CPC title