End-to-end text-to-speech conversion

Patent metadata

Field	Value
Publication number	US-2020098350-A1
Application number	US-201916696101-A
Country	US
Kind code	A1
Filing date	Nov 26, 2019
Priority date	Mar 29, 2017
Publication date	Mar 26, 2020
Grant date	—

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for generating speech from text. One of the systems includes one or more computers and one or more storage devices storing instructions that when executed by one or more computers cause the one or more computers to implement: a sequence-to-sequence recurrent neural network configured to: receive a sequence of characters in a particular natural language, and process the sequence of characters to generate a spectrogram of a verbal utterance of the sequence of characters in the particular natural language; and a subsystem configured to: receive the sequence of characters in the particular natural language, and provide the sequence of characters as input to the sequence-to-sequence recurrent neural network to obtain as output the spectrogram of the verbal utterance of the sequence of characters in the particular natural language.

First claim

1 - 20 . (canceled)

21 . A computer-implemented method for generating, from a sequence of characters in a particular natural language, a spectrogram of a verbal utterance of the sequence of characters in the particular natural language using a text-to-speech conversion system, the method comprising:
processing, using an encoder neural network of the text-to-speech conversion system, the sequence of characters to generate a respective encoded representation of each of the characters in the sequence; receiving a sequence of decoder inputs; for each decoder input in the sequence of decoder inputs, processing, using a decoder neural network of the text-to-speech conversion system, the decoder input and the encoded representations to generate multiple frames of the spectrogram; and generating a waveform from the spectrogram of the verbal utterance of the sequence of characters in the particular natural language.

22 . The method of claim 21 , wherein the encoder neural network comprises an encoder pre-net neural network and an encoder CBHG neural network, and
wherein processing, using the encoder neural network of the text-to-speech conversion system, the sequence of characters to generate a respective encoded representation of each of the characters in the sequence comprises:
receiving, using the encoder pre-net neural network, a respective embedding of each character in the sequence,
processing, using the encoder pre-net neural network, the respective embedding of each character in the sequence to generate a respective transformed embedding of the character, and
processing, using the encoder CBHG neural network, a respective transformed embedding of each character in the sequence to generate a respective encoded representation of the character.

23 . The method of claim 22 , wherein the encoder CBHG neural network comprises a bank of 1-D convolutional filters, followed by a highway network, and followed by a bidirectional recurrent neural network.)

24 . The method of claim 23 , wherein the bidirectional recurrent neural network is a gated recurrent unit neural network.

25 . The method of claim 23 , wherein the encoder CBHG includes a residual connection between the transformed embeddings and outputs of the bank of 1-D convolutional filters.

26 . The method of claim 23 , wherein the bank of 1-D convolutional filters includes a max pooling along time layer with stride one.

27 . The method of claim 21 , wherein a first decoder input in the sequence is a predetermined initial frame.

28 . The method of claim 21 , wherein the spectrogram is a compressed spectrogram.

29 . The method of claim 28 , wherein the compressed spectrogram is a mel-scale spectrogram.

30 . The method of claim 28 , further comprising:
processing the compressed spectrogram to generate a waveform synthesizer input; and processing, using a waveform synthesizer of the text-to-speech conversion system, the waveform synthesizer input to generate the waveform of the verbal utterance of the input sequence of characters in the particular natural language.

31 . The method of claim 21 , further comprising:
generating speech using the waveform; and providing the generated speech for playback.

32 . The method of claim 30 , wherein the waveform synthesizer is a trainable spectrogram to waveform inverter.

33 . The method of claim 30 , wherein the waveform synthesizer is a vocoder.

34 . The method of claim 30 , wherein the waveform synthesizer input is a linear-scale spectrogram of the verbal utterance of the input sequence of characters in the particular natural language.

35 . One or more non-transitory computer storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform operations for generating, from a sequence of characters in a particular natural language, a spectrogram of a verbal utterance of the sequence of characters in the particular natural language using a text-to-speech conversion system, the operations comprising:
processing, using an encoder neural network of the text-to-speech conversion system, the sequence of characters to generate a respective encoded representation of each of the characters in the sequence; receiving a sequence of decoder inputs; for each decoder input in the sequence of decoder inputs, processing, using a decoder neural network of the text-to-speech conversion system, the decoder input and the encoded representations to generate multiple frames of the spectrogram; and generating a waveform from the spectrogram of the verbal utterance of the sequence of characters in the particular natural language.

36 . The one or more non-transitory computer storage media of claim 35 , wherein the encoder neural network comprises an encoder pre-net neural network and an encoder CBHG neural network, and
wherein processing, using the encoder neural network of the text-to-speech conversion system, the sequence of characters to generate a respective encoded representation of each of the characters in the sequence comprises:
receiving, using the encoder pre-net neural network, a respective embedding of each character in the sequence,
processing, using the encoder pre-net neural network, the respective embedding of each character in the sequence to generate a respective transformed embedding of the character, and
processing, using the encoder CBHG neural network, a respective transformed embedding of each character in the sequence to generate a respective encoded representation of the character.

37 . The one or more non-transitory computer storage media of claim 35 , wherein the spectrogram is a compressed spectrogram.

38 . The one or more non-transitory computer storage media of claim 35 , wherein the spectrogram is a mel-scale spectrogram.

39 . The one or more non-transitory computer storage media of claim 37 , further comprising:
processing the compressed spectrogram to generate a waveform synthesizer input; and processing, using a waveform synthesizer of the text-to-speech conversion system, the waveform synthesizer input to generate the waveform of the verbal utterance of the input sequence of characters in the particular natural language.

40 . The one or more non-transitory computer storage media of claim 35 , further comprising:
generating speech using the waveform; and providing the generated speech for playback.

Assignees

• Google Llc

Inventors

• Bengio Samuel
• Wang Yuxuan
• Yang Zongheng
• Chen Zhifeng
• Wu Yonghui
• Agiomyrgiannakis Ioannis
• Weiss Ron J
• Jaitly Navdeep
• Rifkin Ryan M
• Clark Robert Andrew James
• Le Quoc V
• Ryan Russell J
• Xiao Ying

Classifications

• G10L15/16 Primary

  using artificial neural networks · CPC title

• G10L13/04

  Details of speech synthesis systems, e.g. synthesiser structure or memory management · CPC title

• G06N3/084

  Backpropagation, e.g. using gradient descent · CPC title

• G10L13/08 Primary

  Text analysis or generation of parameters for speech synthesis out of text, e.g. grapheme to phoneme translation, prosody generation or stress or intonation determination · CPC title

• G10L25/30

  using neural networks · CPC title