Generating diverse and natural text-to-speech samples

Patent metadata

Field	Value
Publication number	US-11475874-B2
Application number	US-202117163007-A
Country	US
Kind code	B2
Filing date	Jan 29, 2021
Priority date	Jan 29, 2021
Publication date	Oct 18, 2022
Grant date	Oct 18, 2022

Abstract

A method of generating diverse and natural text-to-speech (TTS) samples includes receiving a text and generating a speech sample based on the text using a TTS model. A training process trains the TTS model to generate the speech sample by receiving training samples. Each training sample includes a spectrogram and a training text corresponding to the spectrogram. For each training sample, the training process identifies speech units associated with the training text. For each speech unit, the training process generates a speech embedding, aligns the speech embedding with a portion of the spectrogram, extracts a latent feature from the aligned portion of the spectrogram, and assigns a quantized embedding to the latent feature. The training process generates the speech sample by decoding a concatenation of the speech embeddings and a quantized embeddings for the speech units associated with the training text corresponding to the spectrogram.

First claim

What is claimed is:

1. A computer-implemented method when executed by data processing hardware causes the data processing hardware to perform operations comprising:
receiving a sequence of text; and
generating one or more speech samples based on the sequence of text using a text-to-speech (TTS) model, the TTS model trained to generate the one or more speech samples by:
receiving a plurality of training samples, each training sample of the plurality of training samples comprising a reference spectrogram and a training sequence of text corresponding to the reference spectrogram; and
for each training sample of the plurality of training samples:
identifying a sequence of speech units associated with the training sequence of text corresponding to the reference spectrogram;
for each speech unit of the sequence of speech units:
generating a speech embedding;
aligning the speech embedding with a corresponding portion of the reference spectrogram;
extracting a latent feature from the corresponding portion of the reference spectrogram aligned with the speech embedding; and
assigning a quantized embedding to the latent feature from the corresponding portion of the reference spectrogram aligned with the speech embedding; and

decoding a concatenation of the speech embeddings and the quantized embeddings for the sequence of speech units associated with the training sequence of text corresponding to the reference spectrogram.

2. The method of claim 1 , wherein assigning the quantized embedding to the latent feature from the corresponding portion of the reference spectrogram aligned with the speech embedding comprises assigning the quantized embedding to the latent feature from the corresponding portion of the reference spectrogram aligned with the speech embedding after sampling the quantized embedding from a posterior distribution in a continuous latent space.

3. The method of claim 1 , wherein the operations further comprise:
for each of a plurality of available quantized embeddings, determining a Euclidean distance between the corresponding quantized embedding and the latent feature extracted from the corresponding portion of the reference spectrogram aligned with the speech embedding; and
selecting, from among the plurality of available quantized embeddings, the quantized embedding associated with a minimum Euclidean distance between the corresponding quantized embedding and the latent feature extracted from the corresponding portion of the reference spectrogram aligned with the speech embedding,
wherein assigning the quantized embedding to the latent feature comprises assigning the selected quantized embedding associated with the minimum Euclidean distance.

4. The method of claim 1 , wherein the operations further comprise, during training of the TTS model:
determining a reconstruction loss represented as a likelihood probability based on the speech embedding and the latent feature;
determining a KL divergence between a prior distribution of the latent feature and a posterior distribution of the latent feature; and
optimizing an evidence lower bound (ELBO) loss of the TTS model based on a difference between the determined reconstruction loss and the determined KL divergence.

5. The method of claim 1 , wherein the operations further comprise, during training the TTS model:
generating, as output from the TTS model, a posterior distribution for the plurality of training samples; and
training an autoregressive (AR) prior neural network to generate a prior distribution by fitting the generated posterior distribution over the plurality of training samples.

6. The method of claim 5 , wherein generating the one or more speech samples is based on sampling the prior distribution from the trained AR prior neural network.

7. The method of claim 1 , wherein the sequence speech units comprises a sequence of phonemes.

8. The method of claim 1 , wherein the latent feature comprises a latent prosody feature.

9. The method of claim 1 , wherein the one or more speech samples comprise a first speech sample and a second speech sample, the first speech sample having different speech attributes than the second speech sample.

10. The method of claim 1 , wherein the operations further comprise training a speech recognition model using the one or more speech samples generated by the TTS model.

11. A system comprising:
data processing hardware; and
memory hardware in communication with the data processing hardware, the memory hardware storing instructions that when executed on the data processing hardware cause the data processing hardware to perform operations comprising:
receiving a sequence of text; and
generating one or more speech samples based on the sequence of text using a text-to-speech (TTS) model, the TTS model trained to generate the one or more speech samples by:
receiving a plurality of training samples, each training sample of the plurality of training samples comprising a reference spectrogram and a training sequence of text corresponding to the reference spectrogram; and
for each training sample of the plurality of training samples:
identifying a sequence of speech units associated with the training sequence of text corresponding to the reference spectrogram;
for each speech unit of the sequence of speech units:
 generating a speech embedding;
 aligning the speech embedding with a corresponding portion of the reference spectrogram;
 extracting a latent feature from the corresponding portion of the reference spectrogram aligned with the speech embedding; and
 assigning a quantized embedding to the latent feature from the corresponding portion of the reference spectrogram aligned with the speech embedding; and
decoding a concatenation of the speech embeddings and the quantized embeddings for the sequence of speech units associated with the training sequence of text corresponding to the reference spectrogram.

12. The system of claim 11 , wherein assigning the quantized embedding to the latent feature from the corresponding portion of the reference spectrogram aligned with the speech embedding comprises assigning the quantized embedding to the latent feature from the corresponding portion of the reference spectrogram aligned with the speech embedding after sampling the quantized embedding from a posterior distribution in a continuous latent space.

13. The system of claim 11 , wherein the operations further comprise:
for each of a plurality of available quantized embeddings, determining a Euclidean distance between the corresponding quantized embedding and the latent feature extracted from the corresponding portion of the reference spectrogram aligned with the speech embedding; and
selecting, from among the plurality of available quantized embeddings, the quantized embedding associated with a minimum Euclidean distance between the corresponding quantized embedding and the latent feature extracted from the corresponding portion of the reference spectrogram aligned with the speech embedding,
wherein assigning the quantized embedding to the latent feature comprises assigning the selected quantized embedding associated with the minimum Euclidean distance.

14. The method of claim 1 , wherein the operations further comprise, during training of the TTS model:
determining a reconstruction loss represented as a likelihood probability based on the speech embedding and the latent feature;
determining a KL divergence between a prior distribution of the latent feature and a posterior distribution of the latent feature; and
optimizing an evidence lower bound (ELBO) loss of the TTS model based on a difference between the

Assignees

• Google Llc

Inventors

• Zhang Yu
• Ramabhadran Bhuvana
• Rosenberg Andrew
• Wu Yonghui
• Chun Byungha
• Weiss Ron
• Cao Yuan

Classifications

• G10L13/10

  Prosody rules derived from text; Stress or intonation · CPC title

• G06N3/08

  Learning methods · CPC title

• G10L13/06

  Elementary speech units used in speech synthesisers; Concatenation rules · CPC title

• G10L15/063

  Training · CPC title

• G10L13/08 Primary

  Text analysis or generation of parameters for speech synthesis out of text, e.g. grapheme to phoneme translation, prosody generation or stress or intonation determination · CPC title