End-to-end multi-talker overlapping speech recognition

Patent metadata

Field	Value
Publication number	US-12266347-B2
Application number	US-202218055553-A
Country	US
Kind code	B2
Filing date	Nov 15, 2022
Priority date	May 1, 2020
Publication date	Apr 1, 2025
Grant date	Apr 1, 2025

Abstract

A method for training a speech recognition model with a loss function includes receiving an audio signal including a first segment corresponding to audio spoken by a first speaker, a second segment corresponding to audio spoken by a second speaker, and an overlapping region where the first segment overlaps the second segment. The overlapping region includes a known start time and a known end time. The method also includes generating a respective masked audio embedding for each of the first and second speakers. The method also includes applying a masking loss after the known end time to the respective masked audio embedding for the first speaker when the first speaker was speaking prior to the known start time, or applying the masking loss prior to the known start time when the first speaker was speaking after the known end time.

First claim

What is claimed is:

1. A computer-implemented method when executed on data processing hardware causes the data processing hardware to perform operations comprising:
receiving a sequence of acoustic frames characterizing a speech conversation between two or more speakers;
encoding, by an audio encoder of a speech recognition model, the sequence of acoustic frames into corresponding audio embeddings;
for each corresponding audio embedding:
receiving a speaker embedding associated with the corresponding audio embedding;
identifying a respective speaker among the two or more speakers that is associated with the corresponding audio embedding based on the speaker embedding;
concatenating the corresponding audio embedding with the speaker embedding; and
generating, using a masking model, a masked embedding corresponding to the identified respective speaker based on the corresponding audio embedding concatenated with the speaker embedding; and

for each respective speaker among the two or more speakers, generating, by a dedicated speaker branch of the speech recognition model for the respective speaker that receives each masked embedding corresponding to the respective speaker, a respective transcription that transcribes a respective segment of the speech conversation spoken by the respective speaker,
wherein a training process trains the masking model and the each of the dedicated speaker branches by:
in a first stage, training a single recurrent neural network-transducer (RNN-T) model using training examples;
dividing the single RNN-T model into each of the dedicated speaker branches; and
in a second stage, training the masking model and fine-tuning each of the dedicated speaker branches using the training examples by applying a respective masking loss to a segment of each training example of the training examples where a speaker is not speaking to minimize an RNN-T loss for each dedicated speaker branch of the masking model.

2. The method of claim 1 , wherein the speech recognition model comprises a recurrent neural network-transducer (RNN-T) architecture.

3. The method of claim 2 , wherein the dedicated speaker branch of the speech recognition model for the respective speaker comprises:
a dedicated label encoder for the respective speaker configured to:
receive, as input, sequences of non-blank symbols output by a dedicated final softmax layer for the respective speaker; and
generate labels for the respective speaker; and

a dedicated joint network for the respective speaker configured to:
receive, as input, the labels for the respective speaker generated by the dedicated label encoder and each masked embedding corresponding to the respective speaker; and
generate, at each of a plurality of output steps, a probability distribution over possible speech recognition hypotheses for the respective speaker.

4. The method of claim 3 , wherein generating the respective transcription that transcribes the respective segment of the speech conversation spoken by the respective speaker is based on the probability distribution over possible speech recognition hypotheses generated at each of the plurality of output steps for the respective speaker.

5. The method of claim 1 , wherein the operations further comprise:
for each corresponding audio embedding, determining a fixed input assigned to the respective speaker associated with the corresponding audio embedding,
wherein identifying the respective speaker among the two or more speakers that is associated with the corresponding audio frame is based on the fixed input.

6. The method of claim 1 , wherein the operations further comprise generating a transcript of the speech conversation between the two or more speakers based on the respective transcription generated for each respective speaker among the two or more speakers.

7. The method of claim 1 , wherein the operations further comprise displaying the transcription on a display screen in communication with the data processing hardware.

8. The method of claim 1 , wherein:
wherein the sequence of acoustic frames characterizing the speech conversation are segmented from a monophonic signal captured by a user device; and
the data processing hardware resides on the user device.

9. An automated speech recognition (ASR) model comprising:
an audio encoder configured to:
receive, as input, a sequence of acoustic frames characterizing a speech conversation between two or more speakers; and
generate, at each of a plurality of output steps, an audio embedding corresponding to each acoustic frame in the sequence of acoustic frames;

a masking model configured to:
receive, as input, the audio embedding generated by the audio encoder at each of the plurality of output steps;
receive, as input, a speaker embedding associated with each corresponding audio embedding generated by the audio encoder at each of the plurality of output steps, the speaker embedding identifying a respective speaker among the two or more speakers that is associated with the corresponding audio embedding; and
generate, at each of the plurality of output steps, a respective masked embedding for the respective speaker among the two or more speakers that is associated with the audio embedding generated by the audio encoder at the corresponding output step, the respective masked embedding based on a concatenation of the speaker embedding that identifies the respective speaker among the two or more speakers that is associated with the corresponding audio embedding at the corresponding output step; and

for each respective speaker among the two or more speakers, a dedicated speaker branch configured to:
receive, as input, each respective masked embedding generated by the masking model for the respective speaker; and
generate a respective transcription that transcribes a respective segment of the speech conversation spoken by the respective speaker,
wherein a training process trains the masking model and the each of the dedicated speaker branches by:
in a first stage, training a single recurrent neural network-transducer (RNN-T) model using training examples;
dividing the single RNN-T model into each of the dedicated speaker branches; and
in a second stage, training the masking model and fine-tuning each of the dedicated speaker branches using the training examples by applying a respective masking loss to a segment of each training example of the training examples where a speaker is not speaking to minimize an RNN-T loss for each dedicated speaker branch of the masking model.

10. The ASR model of claim 9 , wherein the speech recognition model comprises a recurrent neural network-transducer (RNN-T) architecture.

11. The ASR model of claim 10 , wherein the dedicated speaker branch for each respective speaker comprises:
a dedicated label encoder for the respective speaker configured to:
receive, as input, sequences of non-blank symbols output by a dedicated final softmax layer for the respective speaker; and
generate labels for the respective speaker; and

a dedicated joint network for the respective speaker configured to:
receive, as input, the labels for the respective speaker generated by the dedicated label encoder and each respective masked embedding generated by the masking model for the respective speaker; and
generate, at each of a plurality of output steps, a probability distribution over possible speech recognition hypotheses for the respective speaker.

12. The ASR model of claim 11 , wherein the dedicated speaker branch for the respective speaker generates the respective transcription that transcribes the respective segment of the speech conversation spoken by the respective

Assignees

• Google Llc

Inventors

• Tripathi Anshuman
• Lu Han
• Sak Hasim

Classifications

• G06N3/0442

  characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU] · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06N3/0455

  Auto-encoder networks; Encoder-decoder networks · CPC title

• G10L15/16

  using artificial neural networks · CPC title

• G10L15/04

  Segmentation; Word boundary detection · CPC title