Parallel tacotron non-autoregressive and controllable TTS

Patent metadata

Field	Value
Publication number	US-11908448-B2
Application number	US-202117327076-A
Country	US
Kind code	B2
Filing date	May 21, 2021
Priority date	Oct 21, 2020
Publication date	Feb 20, 2024
Grant date	Feb 20, 2024

Abstract

A method for training a non-autoregressive TTS model includes receiving training data that includes a reference audio signal and a corresponding input text sequence. The method also includes encoding the reference audio signal into a variational embedding that disentangles the style/prosody information from the reference audio signal and encoding the input text sequence into an encoded text sequence. The method also includes predicting a phoneme duration for each phoneme in the input text sequence and determining a phoneme duration loss based on the predicted phoneme durations and a reference phoneme duration. The method also includes generating one or more predicted mel-frequency spectrogram sequences for the input text sequence and determining a final spectrogram loss based on the predicted mel-frequency spectrogram sequences and a reference mel-frequency spectrogram sequence. The method also includes training the TTS model based on the final spectrogram loss and the corresponding phoneme duration loss.

First claim

What is claimed is:

1. A computer-implemented method when executed on data processing hardware causes the data processing hardware to perform operations for training a non-autoregressive text-to-speech (TTS) model, the operations comprising:
receiving training data including a reference audio signal and a corresponding input text sequence, the reference audio signal comprising a spoken utterance and the input text sequence corresponds to a transcript of the reference audio signal;
encoding, using a residual encoder, the reference audio signal into a variational embedding, the variational embedding disentangling style/prosody information from the reference audio signal;
encoding, using a text encoder, the input text sequence into an encoded text sequence;
predicting, using a duration decoder comprising a stack of self-attention blocks followed by two independent projections, based on the encoded text sequence and the variational embedding, a phoneme duration for each phoneme in the input text sequence by:
predicting, using a sigmoid activation following a first one of the two independent projections, a probability of non-zero duration for each phoneme;
predicting, using a softplus activation following a second one of the two independent projections, the phoneme duration for each phoneme;
determining whether the probability of non-zero duration predicted for the corresponding phoneme is less than a threshold value; and
when the probability of non-zero duration is less than the threshold value, zeroing out the phoneme duration predicted for the corresponding phoneme;

determining a phoneme duration loss based on the predicted phoneme durations and a reference phoneme duration sampled from the reference audio signal for each phoneme in the input text sequence
generating, as output from a non-autoregressive spectrogram decoder comprising a stack of self-attention blocks, based on an output of the duration decoder, multiple predicted mel-frequency spectrogram sequences for the input text sequence;
determining a final spectrogram loss based on the multiple predicted mel-frequency spectrogram sequences and a reference mel-frequency spectrogram sequence sampled from the reference audio signal; and
training the TTS model based on the final spectrogram loss and the corresponding phoneme duration loss determined for each phoneme in the input text sequence.

2. The computer-implemented method of claim 1 , wherein each self-attention block in the stack of self-attention blocks comprises an identical lightweight convolution (LConv) block.

3. The computer-implemented method of claim 1 , wherein each self-attention block in the stack of self-attention blocks comprises an identical transformer block.

4. The computer-implemented method of claim 1 , wherein the input text sequence comprises words each having one or more phonemes, silences at all word boundaries, and punctuation marks.

5. The computer-implemented method of claim 1 , wherein:
the residual encoder comprises a global variational autoencoder (VAE); and
encoding the reference audio signal into the variational embedding comprises:
sampling the reference mel-frequency spectrogram sequence from the reference audio signal; and
encoding, using the global VAE, the reference mel-frequency spectrogram sequence into the variational embedding.

6. The computer-implemented method of claim 1 , wherein:
the residual encoder comprises a phoneme-level fine-grained variational autoencoder (VAE); and
encoding the reference audio signal into the variational embedding comprises:
sampling the reference mel-frequency spectrogram sequence from the reference audio signal;
aligning the reference mel-frequency spectrogram sequence with each phoneme in a sequence of phonemes extracted from the input text sequence; and
encoding, using the phoneme-level fine-grained VAE, based on aligning the reference mel-frequency spectrogram sequence with each phoneme in the sequence of phonemes, a sequence of phoneme-level variational embeddings.

7. The computer-implemented method of claim 1 , wherein the residual encoder comprises a stack of lightweight convolution (LConv) blocks, each LConv block in the stack of LConv blocks comprises:
a gated linear unit (GLU) layer;
a LConv layer configured to receive an output of the GLU layer;
a residual connection configured to concatenate an output of the LConv layer with an input to the GLU layer; and
a final feedforward layer configured to receive, as input, the residual connection concatenating the output of the LConv layer with the input to the GLU layer.

8. The computer-implemented method of claim 1 , wherein the operations further comprise:
upsampling, using the reference phoneme duration sampled from the reference audio signal for each phoneme in the input text sequence, the output of the duration decoder into a number of frames; and
obtaining positional embeddings representing phoneme position information for each phoneme in the input text utterance,
wherein generating the multiple predicted mel-frequency spectrogram sequences for the input text sequence is based on the positional embeddings and the upsampling of the output of the duration decoder into the number of frames.

9. The computer-implemented method of claim 1 , wherein:
generating the multiple predicted mel-frequency spectrogram sequences for the input text sequence comprises generating, as output from each self-attention block in the stack of self-attention blocks of the spectrogram decoder, a respective mel-frequency spectrogram sequence; and
determining the final spectrogram loss comprises:
for each respective predicted mel-frequency spectrogram sequence, determining a respective spectrogram loss based on the predicted mel-frequency spectrogram sequence and the reference mel-frequency spectrogram sequence; and
aggregating the respective spectrogram losses determined for the predicted mel-frequency spectrogram sequences to generate the final spectrogram loss.

10. The computer-implemented method of claim 1 , wherein the operations further comprise:
concatenating the encoded text sequence, the variational embedding, and a reference speaker embedding representing an identity of a reference speaker that uttered the reference audio signal; and
generating the output of the duration decoder based on the duration decoder receiving, as input, the concatenation of the encoded text sequence, the variational embedding, and the reference speaker embedding.

11. The computer-implemented method of claim 1 , wherein:
the input text sequence includes a sequence of phonemes; and
encoding the input text sequence into the encoded text sequence comprises:
receiving, from a phoneme look-up table, a respective embedding of each phoneme in the sequence of phonemes;
for each phoneme in the sequence of phonemes, processing, using an encoder pre-net neural network of the text encoder, the respective embedding to generate a respective transformed embedding of the phoneme;
processing, using a bank of convolutional blocks, the respective transformed embeddings to generate convolution outputs; and
processing, using a stack of self-attention blocks, the convolution outputs to generate the encoded text sequence.

12. A system for training a non-autoregressive text-to-speech (TTS) model, the system comprising:
data processing hardware; and
memory hardware in communication with the data processing hardware, the memory hardware storing instructions that when executed by the data processing hardware cause the data processing hardware to perform operations comprising:
receiving training data including a reference audio signal and a correspondin

Assignees

• Google Llc

Inventors

• Elias Isaac
• Shen Jonathan
• Zhang Yu
• Jia Ye
• Weiss Ron J
• Wu Yonghui
• Chun Byungha

Classifications

• G06N3/0455

  Auto-encoder networks; Encoder-decoder networks · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/09

  Supervised learning · CPC title

• G10L13/08 Primary

  Text analysis or generation of parameters for speech synthesis out of text, e.g. grapheme to phoneme translation, prosody generation or stress or intonation determination · CPC title

• G06F40/126

  Character encoding · CPC title