Unsupervised parallel tacotron non-autoregressive and controllable text-to-speech

Patent metadata

Field	Value
Publication number	US-11823656-B2
Application number	US-202117326542-A
Country	US
Kind code	B2
Filing date	May 21, 2021
Priority date	Mar 22, 2021
Publication date	Nov 21, 2023
Grant date	Nov 21, 2023

Abstract

A method for training a non-autoregressive TTS model includes obtaining a sequence representation of an encoded text sequence concatenated with a variational embedding. The method also includes using a duration model network to predict a phoneme duration for each phoneme represented by the encoded text sequence. Based on the predicted phoneme durations, the method also includes learning an interval representation and an auxiliary attention context representation. The method also includes upsampling, using the interval representation and the auxiliary attention context representation, the sequence representation into an upsampled output specifying a number of frames. The method also includes generating, based on the upsampled output, one or more predicted mel-frequency spectrogram sequences for the encoded text sequence. The method also includes determining a final spectrogram loss based on the predicted mel-frequency spectrogram sequences and a reference mel-frequency spectrogram sequence and training the TTS model based on the final spectrogram loss.

First claim

What is claimed is:

1. A computer-implemented method when executed on data processing hardware causes the data processing hardware to perform operations for training a non-autoregressive text-to-speech (TTS) model, the operations comprising:
obtaining a sequence representation of an encoded text sequence concatenated with a variational embedding;
using a duration model network:
predicting, based on the sequence representation, a phoneme duration for each phoneme represented by the encoded text sequence;
based on the predicted phoneme durations:
learning, using a first function conditioned on the sequence representation, an interval representation matrix; and
learning, using a second function conditioned on the sequence representation, an auxiliary attention context representation;

determining a product of the interval representation matrix and the sequence representation;
determining an Einstein summation (einsum) of the interval representation matrix and the auxiliary attention context representation; and
upsampling, based on summing the product of the interval representation matrix and the sequence representation and a projection of the einsum, the sequence representation into an upsampled output specifying a number of frames;

generating, as output from a spectrogram decoder comprising a stack of one or more self-attention blocks, based on the upsampled output, one or more predicted mel-frequency spectrogram sequences for the encoded text sequence;
determining a final spectrogram loss based on the one or more predicted mel-frequency spectrogram sequences and a reference mel-frequency spectrogram sequence; and
training the TTS model based on the final spectrogram loss.

2. The computer-implemented method of claim 1 , wherein the first function and the second function each comprise a respective multi-layer perception-based learnable function.

3. The computer-implemented method of claim 1 , wherein the operations further comprise:
determining a global phoneme duration loss based on the predicted phoneme durations and an average phoneme duration,
wherein training the TTS model is further based on the global phoneme duration loss.

4. The computer implemented method of claim 3 , wherein training the TTS model based on the final spectrogram loss and the global phoneme duration loss comprises training the duration model network to predict the phoneme duration for each phoneme without using supervised phoneme duration labels extracted from an external aligner.

5. The computer-implemented method of claim 1 , wherein the operations further comprise, using the duration model network:
based on the predicted phoneme durations, generating, for each phoneme represented by the encoded text sequence, respective start and end boundaries;
mapping, based on a number of phonemes represented by the encoded text sequence and a number of reference frames in the reference mel-frequency spectrogram sequence, the respective start and end boundaries generated for each phoneme into respective grid matrices,
wherein learning the interval representation is based on the respective grid matrices mapped from the start and end boundaries; and
wherein learning the auxiliary attention context representation is based on the respective grid matrices mapped from the start and end boundaries.

6. The computer-implemented method of claim 1 , wherein the operations further comprise:
receiving training data including a reference audio signal and a corresponding input text sequence, the reference audio signal comprising a spoken utterance and the input text sequence corresponds to a transcript of the reference audio signal;
encoding, using a residual encoder, the reference audio signal into a variational embedding, the variational embedding disentangling style/prosody information from the reference audio signal; and
encoding, using a text encoder, the input text sequence into the encoded text sequence.

7. The computer-implemented method of claim 6 , wherein:
the residual encoder comprises a global variational autoencoder (VAE); and
encoding the reference audio signal into the variational embedding comprises:
sampling the reference mel-frequency spectrogram sequence from the reference audio signal; and
encoding, using the global VAE, the reference mel-frequency spectrogram sequence into the variational embedding.

8. The computer-implemented method of claim 6 , wherein:
the residual encoder comprises a phoneme-level fine-grained variational autoencoder (VAE); and
encoding the reference audio signal into the variational embedding comprises:
sampling the reference mel-frequency spectrogram sequence from the reference audio signal;
aligning the reference mel-frequency spectrogram sequence with each phoneme in a sequence of phonemes extracted from the input text sequence; and
encoding, using the phoneme-level fine-grained VAE, based on aligning the reference mel-frequency spectrogram sequence with each phoneme in the sequence of phonemes, a sequence of phoneme-level variational embeddings.

9. The computer-implemented method of claim 6 , wherein the residual encoder comprises a stack of lightweight convolution (LConv) blocks, each LConv block in the stack of LConv blocks comprises:
a gated linear unit (GLU) layer;
a LConv layer configured to receive an output of the GLU layer;
a residual connection configured to concatenate an output of the LConv layer with an input to the GLU layer; and
a final feedforward layer configured to receive, as input, the residual connection concatenating the output of the LConv layer with the input to the GLU layer.

10. The computer-implemented method of claim 6 , wherein the operations further comprise:
concatenating the encoded text sequence, the variational embedding, and a reference speaker embedding representing an identity of a reference speaker that uttered the reference audio signal; and
generating the sequence representation based on the duration modeling network receiving, as input, the concatenation of the encoded text sequence, the variational embedding, and the reference speaker embedding.

11. The computer-implemented method of claim 6 , wherein:
the input text sequence includes a sequence of phonemes; and
encoding the input text sequence into the encoded text sequence comprises:
receiving, from a phoneme look-up table, a respective embedding of each phoneme in the sequence of phonemes;
for each phoneme in the sequence of phonemes, processing, using an encoder pre-net neural network of the text encoder, the respective embedding to generate a respective transformed embedding of the phoneme;
processing, using a bank of convolutional blocks, the respective transformed embeddings to generate convolution outputs; and
processing, using a stack of self-attention blocks, the convolution outputs to generate the encoded text sequence.

12. The computer-implemented method of claim 1 , wherein each self-attention block in the stack of self-attention blocks comprises an identical lightweight convolution (LConv) block.

13. The computer-implemented method of claim 1 , wherein each self-attention block in the stack of self-attention blocks comprises an identical transformer block.

14. A system for training a non-autoregressive text-to-speech (TTS) model, the system comprising:
data processing hardware; and
memory hardware in communication with the data processing hardware, the memory hardware storing instructions that when executed by the data processing hardware cause the data processing hardware to perform operations comprising:
obtaining a sequence representation of an encoded text sequence c

Assignees

• Google Llc

Inventors

• Elias Isaac
• Chun Byungha
• Shen Jonathan
• Jia Ye
• Zhang Yu
• Wu Yonghui

Classifications

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/0475

  Generative networks · CPC title

• G06N3/0895

  Weakly supervised learning, e.g. semi-supervised or self-supervised learning · CPC title

• G06N3/0455

  Auto-encoder networks; Encoder-decoder networks · CPC title

• G10L13/08 Primary

  Text analysis or generation of parameters for speech synthesis out of text, e.g. grapheme to phoneme translation, prosody generation or stress or intonation determination · CPC title