Speech recognition using unspoken text and speech synthesis

Patent metadata

Field	Value
Publication number	US-11837216-B2
Application number	US-202318168969-A
Country	US
Kind code	B2
Filing date	Feb 14, 2023
Priority date	May 7, 2020
Publication date	Dec 5, 2023
Grant date	Dec 5, 2023

Abstract

A method for training a generative adversarial network (GAN)-based text-to-speech (TTS) model and a speech recognition model in unison includes obtaining a plurality of training text utterances. At each of a plurality of output steps for each training text utterance, the method also includes generating, for output by the GAN-Based TTS model, a synthetic speech representation of the corresponding training text utterance, and determining, using an adversarial discriminator of the GAN, an adversarial loss term indicative of an amount of acoustic noise disparity in one of the non-synthetic speech representations selected from the set of spoken training utterances relative to the corresponding synthetic speech representation of the corresponding training text utterance. The method also includes updating parameters of the GAN-based TTS model based on the adversarial loss term determined at each of the plurality of output steps for each training text utterance of the plurality of training text utterances.

First claim

What is claimed is:

1. A computer-implemented method executed on data processing hardware that causes the data processing hardware to perform operations comprising:
obtaining a spoken training utterance comprising a corresponding transcription paired with a corresponding non-synthetic speech representation of the spoken training utterance;
obtaining an embedding representing speaker characteristics of a speaker that spoke the spoken training utterance;
conditioning the corresponding transcription of the spoken training utterance on the embedding representing the speaker characteristics of the speaker that spoke the spoken training utterance;
generating, as output from a text-to-speech (TTS) model configured to receive the corresponding transcription of the spoken training utterance as input, a synthetic speech representation of the spoken training utterance conditioned on the embedding; and
training a speech recognition model on the non-synthetic speech representation of the spoken training utterance and the synthetic speech representation generated as output from the TTS model.

2. The computer-implemented method of claim 1 , wherein obtaining the embedding comprises extracting, from the non-synthetic speech representation of the spoken training utterance, the embedding representing the speaker characteristics of the speaker that spoke the spoken training utterance.

3. The computer-implemented method of claim 1 , wherein the synthetic speech representation generated as output from the TTS model is represented by a sequence of mel-frequency spectrogram frames.

4. The computer-implemented method of claim 1 , wherein the non-synthetic speech representation of the spoken training utterance is represented by a sequence of mel-frequency spectrogram frames.

5. The computer-implemented method of claim 1 , wherein the operations further comprise applying data augmentation to the synthetic speech representation output from the TTS model.

6. The computer-implemented method of claim 1 , wherein the data augmentation applied to the synthetic speech representation comprises spectrum augmentation.

7. The computer-implemented method of claim 1 , wherein the speech recognition model comprises a frame alignment-based transducer model.

8. The computer-implemented method of claim 7 , wherein the speech recognition model comprising the frame alignment-based transducer model comprises a Recurrent Neural Network-Transducer (RNN-T) model.

9. The computer-implemented method of claim 1 , wherein the TTS model comprises:
an encoder neural network configured to:
receive, as input, the transcription as a sequence of phonemes; and
generate, as output, a sequence of context vectors; and

a decoder neural network configured to:
receive, as input, each context vector in the sequence of context vectors generated as output by the encoder neural network; and
generate, as output for each context vector, a corresponding frame in a sequence of mel-frequency spectrogram frames.

10. The method of claim 1 , wherein the operations further comprise, while training the speech recognition model:
at each of a plurality of output steps for the synthetic speech representation generated as output from the TTS model:
determining, for output by the speech recognition model, a first probability distribution over possible synthetic speech recognition hypotheses for the synthetic speech representation; and
generating a synthetic speech loss term based on the first probability distribution over possible synthetic speech recognition hypotheses for the synthetic speech representation and the transcription of the spoken training utterance from which the synthetic speech representation is generated; and

at each of a plurality of output steps for the non-synthetic speech representation:
determining, for output by the speech recognition model, a second probability distribution over possible non-synthetic speech recognition hypotheses for the non-synthetic speech representation; and
generating a non-synthetic speech loss term based on the second probability distribution over possible non-synthetic speech recognition hypotheses for the non-synthetic speech representation and the transcription that is paired with the non-synthetic speech representation.

11. A system comprising:
data processing hardware; and
memory hardware in communication with the data processing hardware, the memory hardware storing instructions that when executed on the data processing hardware cause the data processing hardware to perform operations comprising:
obtaining a spoken training utterance comprising a corresponding transcription paired with a corresponding non-synthetic speech representation of the spoken training utterance;
obtaining an embedding representing speaker characteristics of a speaker that spoke the spoken training utterance;
conditioning the corresponding transcription of the spoken training utterance on the embedding representing the speaker characteristics of the speaker that spoke the spoken training utterance;
generating, as output from a text-to-speech (TTS) model configured to receive the corresponding transcription of the spoken training utterance as input, a synthetic speech representation of the spoken training utterance conditioned on the embedding; and
training a speech recognition model on the non-synthetic speech representation of the spoken training utterance and the synthetic speech representation generated as output from the TTS model.

12. The system of claim 11 , wherein obtaining the embedding comprises extracting, from the non-synthetic speech representation of the spoken training utterance, the embedding representing the speaker characteristics of the speaker that spoke the spoken training utterance.

13. The system of claim 11 , wherein the synthetic speech representation generated as output from the TTS model is represented by a sequence of mel-frequency spectrogram frames.

14. The system of claim 11 , wherein the non-synthetic speech representation of the spoken training utterance is represented by a sequence of mel-frequency spectrogram frames.

15. The system of claim 11 , wherein the operations further comprise applying data augmentation to the synthetic speech representation output from the TTS model.

16. The system of claim 15 , wherein the data augmentation applied to the synthetic speech representation comprises spectrum augmentation.

17. The system of claim 11 , wherein the speech recognition model comprises a frame alignment-based transducer model.

18. The system of claim 17 , wherein the speech recognition model comprising the frame alignment-based transducer model comprises a Recurrent Neural Network-Transducer (RNN-T) model.

19. The system of claim 11 , wherein the TTS model comprises:
an encoder neural network configured to:
receive, as input, the transcription as a sequence of phonemes; and
generate, as output, a sequence of context vectors; and

a decoder neural network configured to:
receive, as input, each context vector in the sequence of context vectors generated as output by the encoder neural network; and
generate, as output for each context vector, a corresponding frame in a sequence of mel-frequency spectrogram frames.

20. The system of claim 11 , wherein the operations further comprise, while training the speech recognition model:
at each of a plurality of output steps for the synthetic speech representation generated as output from the TTS model:
determining, for output by the speech recognition mod

Assignees

• Google Llc

Inventors

• Chen Zhehuai
• Rosenberg Andrew M
• Ramabhadran Bhuvana
• Mengibar Pedro J Moreno

Classifications

• G10L25/18

  the extracted parameters being spectral information of each sub-band · CPC title

• G10L15/16

  using artificial neural networks · CPC title

• G10L13/00 Primary

  Speech synthesis; Text to speech systems · CPC title

• G10L15/063 Primary

  Training · CPC title

• G10L13/08 Primary

  Text analysis or generation of parameters for speech synthesis out of text, e.g. grapheme to phoneme translation, prosody generation or stress or intonation determination · CPC title