Training for long-form speech recognition

Patent metadata

Field	Value
Publication number	US-12482453-B2
Application number	US-202217935924-A
Country	US
Kind code	B2
Filing date	Sep 27, 2022
Priority date	Oct 5, 2021
Publication date	Nov 25, 2025
Grant date	Nov 25, 2025

Abstract

A method includes obtaining a set of training samples, wherein each training sample includes a corresponding sequence of speech segments corresponding to a training utterance and a corresponding sequence of ground-truth transcriptions for the sequence of speech segments, and wherein each ground-truth transcription includes a start time and an end time of a corresponding speech segment. For each training sample in the set of training samples, the method includes processing, using a speech recognition model, the corresponding sequence of speech segments to obtain one or more speech recognition hypotheses for the training utterance; and, for each speech recognition hypothesis obtained for the training utterance, identifying a respective number of word errors relative to the corresponding sequence of ground-truth transcriptions. The method trains the speech recognition model to minimize word error rate based on the respective number of word errors identified for each speech recognition hypothesis obtained for the training utterance.

First claim

What is claimed is:

1 . A computer-implemented method for training a speech recognition model to recognize long-form speech, the computer-implemented method when executed on data processing hardware causes the data processing hardware to perform operations comprising:
obtaining a set of training samples, each training sample in the set of training samples comprising a corresponding sequence of speech segments corresponding to a training utterance and a corresponding sequence of ground-truth transcriptions for the sequence of speech segments, wherein each ground-truth transcription in the corresponding sequence of ground-truth transcriptions comprises a start time and an end time of a corresponding speech segment; and for each training sample in the set of training samples:
processing, using the speech recognition model, the corresponding sequence of speech segments to obtain an N-best list of speech recognition hypotheses for the training utterance;
for each speech recognition hypothesis in the N-best list of speech recognition hypotheses, identifying a respective number of word errors relative to the corresponding sequence of ground-truth transcriptions;
computing an average of the respective number of word errors identified for each speech recognition hypothesis in the N-best list of speech recognition hypotheses; and
training the speech recognition model to minimize word error rate based on the computed average of the respective number of word errors in the N-best list of speech recognition hypotheses by, for each speech recognition hypothesis in the N-best list of speech recognition hypotheses, one of:
increasing a probability of the speech recognition hypothesis when the respective number of word errors is less than the computed average; or
decreasing the probability of the speech recognition hypothesis when the respective number of word errors is greater than the computed average.

2 . The computer-implemented method of claim 1 , wherein each training sample in the set of training samples further comprises a respective non-speech segment between each pair of adjacent speech segments in the sequence of speech segments.

3 . The computer-implemented method of claim 1 , wherein each speech segment in the sequence of speech segments comprises an average duration of at least six seconds.

4 . The computer-implemented method of claim 1 , wherein the speech recognition model comprises a transducer-based speech recognition model that comprises an encoder network, a prediction network, and a joint network.

5 . The computer-implemented method of claim 4 , wherein the encoder network comprises a plurality of multi-headed attention layers.

6 . The computer-implemented method of claim 4 , wherein:
the prediction network is configured to, at each of a plurality of time steps subsequent to an initial time step:
receive, as input, a sequence of non-blank symbols output by a final softmax layer;
for each non-blank symbol in the sequence of non-blank symbols received as input at the corresponding time step:
generate, using a shared embedding matrix, an embedding of the corresponding non-blank symbol;
assign a respective position vector to the corresponding non-blank symbol; and
weight the embedding proportional to a similarity between the embedding and the respective position vector; and

generate, as output, a single embedding vector at the corresponding time step, the single embedding vector based on a weighted average of the weighted embeddings; and
the joint network is configured to, at each of the plurality of time steps subsequent to the initial time step:
receive, as input, the single embedding vector generated as output from the prediction network at the corresponding time step; and
generate a probability distribution over possible speech recognition hypotheses at the corresponding time step.

7 . The computer-implemented method of claim 6 , wherein the prediction network ties a dimensionality of the shared embedding matrix to a dimensionality of an output layer of the joint network.

8 . A system comprising:
data processing hardware; and memory hardware in communication with the data processing hardware and storing instructions that when executed on the data processing hardware causes the data processing hardware to perform operations comprising:
obtaining a set of training samples, each training sample in the set of training samples comprising a corresponding sequence of speech segments corresponding to a training utterance and a corresponding sequence of ground-truth transcriptions for the sequence of speech segments, wherein each ground-truth transcription in the corresponding sequence of ground-truth transcriptions comprises a start time and an end time of a corresponding speech segment; and
for each training sample in the set of training samples:
processing, using a speech recognition model, the corresponding sequence of speech segments to obtain an N-best list of speech recognition hypotheses for the training utterance;
for each speech recognition hypothesis in the N-best list of speech recognition hypotheses, identifying a respective number of word errors relative to the corresponding sequence of ground-truth transcriptions;
computing an average of the respective number of word errors identified for each speech recognition hypothesis in the N-best list of speech recognition hypotheses; and
training the speech recognition model to minimize word error rate based on the computed average of the respective number of word errors in the N-best list of speech recognition hypotheses by, for each speech recognition hypothesis in the N-best list of speech recognition hypotheses, one of:
increasing a probability of the speech recognition hypothesis when the respective number of word errors is less than the computed average; or
decreasing the probability of the speech recognition hypothesis when the respective number of word errors is greater than the computed average.

9 . The system of claim 8 , wherein each training sample in the set of training samples further comprises a respective non-speech segment between each pair of adjacent speech segments in the sequence of speech segments.

10 . The system of claim 8 , wherein each speech segment in the sequence of speech segments comprises an average duration of at least six seconds.

11 . The system of claim 8 , wherein the speech recognition model comprises a transducer-based speech recognition model that comprises an encoder network, a prediction network, and a joint network.

12 . The system of claim 11 , wherein the encoder network comprises a plurality of multi-headed attention layers.

13 . The system of claim 11 , wherein:
the prediction network is configured to, at each of a plurality of time steps subsequent to an initial time step:
receive, as input, a sequence of non-blank symbols output by a final softmax layer;
for each non-blank symbol in the sequence of non-blank symbols received as input at the corresponding time step:
generate, using a shared embedding matrix, an embedding of the corresponding non-blank symbol;
assign a respective position vector to the corresponding non-blank symbol; and
weight the embedding proportional to a similarity between the embedding and the respective position vector; and

generate, as output, a single embedding vector at the corresponding time step, the single embedding vector based on a weighted average of the weighted embeddings; and
the joint network is configured to, at each of the plurality of time steps subsequent to the initial time step:
receive, as input, the single embedding vector generated as out

Assignees

• Google Llc

Inventors

• Lu Zhiyun
• Doutre Thibault
• Pan Yanwei
• Cao Liangliang
• Prabhavalkar Rohit
• Strohman Trevor
• Zhang Chao

Classifications

• G10L15/1815

  Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning · CPC title

• G10L2015/0636

  Threshold criteria for the updating · CPC title

• G10L15/075

  supervised, i.e. under machine guidance · CPC title

• G10L15/06

  Creation of reference templates; Training of speech recognition systems, e.g. adaptation to the characteristics of the speaker's voice (G10L15/14 takes precedence) · CPC title

• G10L15/22

  Procedures used during a speech recognition process, e.g. man-machine dialogue · CPC title