Text-to-speech synthesis using an autoencoder

Patent metadata

Field	Value
Publication number	US-10249289-B2
Application number	US-201715649311-A
Country	US
Kind code	B2
Filing date	Jul 13, 2017
Priority date	Mar 14, 2017
Publication date	Apr 2, 2019
Grant date	Apr 2, 2019

Abstract

Methods, systems, and computer-readable media for text-to-speech synthesis using an autoencoder. In some implementations, data indicating a text for text-to-speech synthesis is obtained. Data indicating a linguistic unit of the text is provided as input to an encoder. The encoder is configured to output speech unit representations indicative of acoustic characteristics based on linguistic information. A speech unit representation that the encoder outputs is received. A speech unit is selected to represent the linguistic unit, the speech unit being selected from among a collection of speech units based on the speech unit representation output by the encoder. Audio data for a synthesized utterance of the text that includes the selected speech unit is provided.

First claim

What is claimed is:

1. A method performed by one or more computers of a text-to-speech system, the method comprising:
obtaining, by the one or more computers, data indicating a text for text-to-speech synthesis;
providing, by the one or more computers, data indicating a linguistic unit of the text as input to an encoder, the encoder being configured to output speech unit representations indicative of acoustic characteristics based on linguistic information, wherein the encoder is configured to provide speech unit representations learned through machine learning training, wherein the encoder comprises a neural network that was trained as part of an autoencoder network that includes the encoder, a second encoder, and a decoder, wherein:
the encoder is arranged to produce speech unit representations in response to receiving data indicating linguistic units;
the second encoder is arranged to produce speech unit representations in response to receiving data indicating acoustic features of speech units; and
the decoder is arranged to generate output indicating acoustic features of speech units in response to receiving speech unit representations for the speech units from either of the encoder and the second encoder;

receiving, by the one or more computers, a speech unit representation that the encoder outputs in response to receiving the data indicating the linguistic unit as input to the encoder;
selecting, by the one or more computers, a speech unit to represent the linguistic unit, the speech unit being selected from among a collection of speech units based on the speech unit representation output by the encoder; and
providing, by the one or more computers and as output of the text-to-speech system, audio data for a synthesized utterance of the text that includes the selected speech unit.

2. The method of claim 1 , wherein the encoder is configured to provide speech unit representations of a same size to represent speech units having different durations.

3. The method of claim 1 , wherein the encoder is trained to infer speech unit representations from linguistic unit identifiers, wherein the speech unit representations output by the encoder are vectors that have a same fixed length.

4. The method of claim 1 , wherein the encoder comprises a trained neural network having one or more long-short-term memory layers.

5. The method of claim 1 , wherein the encoder, the second encoder, and the decoder are trained jointly; and
wherein the encoder, the second encoder, and the decoder each include one or more long short-term memory layers.

6. The method of claim 1 , wherein the encoder, the second encoder, and the decoder are trained jointly using a cost function configured to minimize:
differences between acoustic features input to the second encoder and acoustic features generated by the decoder; and
differences between the speech unit representations of the encoder and the speech unit representations of the second encoder.

7. The method of claim 1 , further comprising selecting a set of candidate speech units for the linguistic unit based on a vector distances between (i) a first vector that includes the speech unit representation output by the encoder and (ii) second vectors corresponding to speech units in the collection of speech units; and
generating a lattice that includes nodes corresponding to the candidate speech units in the selected set of candidate speech units.

8. The method of claim 7 , wherein selecting the set of candidate speech units comprises:
identifying a predetermined quantity of second vectors that are nearest neighbors for the first vector; and
selecting, as the set of candidate speech units, a set of speech units corresponding to the identified predetermined quantity of second vectors that are nearest neighbors for the first vector.

9. The method of claim 1 , wherein the speech unit representation for the linguistic unit is a first speech unit representation for a first linguistic unit, wherein selecting the speech unit comprises:
obtaining a second speech unit representation for a second linguistic unit that occurs immediately before or after the first linguistic unit in a phonetic representation of the text;
generating a diphone unit representation by concatenating the first speech unit representation with the second speech unit representation; and
selecting, to represent the first linguistic unit, a diphone speech unit identified based on the diphone speech unit representation.

10. A system comprising:
one or more computers; and
one or more data storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform operations comprising:
obtaining, by the one or more computers, data indicating a text for text-to-speech synthesis;
providing, by the one or more computers, data indicating a linguistic unit of the text as input to an encoder, the encoder being configured to output speech unit representations indicative of acoustic characteristics based on linguistic information, wherein the encoder is configured to provide speech unit representations learned through machine learning training, wherein the encoder comprises a neural network that was trained as part of an autoencoder network that includes the encoder, a second encoder, and a decoder, wherein:
the encoder is arranged to produce speech unit representations in response to receiving data indicating linguistic units;
the second encoder is arranged to produce speech unit representations in response to receiving data indicating acoustic features of speech units; and
the decoder is arranged to generate output indicating acoustic features of speech units in response to receiving speech unit representations for the speech units from either of the encoder and the second encoder;

receiving, by the one or more computers, a speech unit representation that the encoder outputs in response to receiving the data indicating the linguistic unit as input to the encoder;
selecting, by the one or more computers, a speech unit to represent the linguistic unit, the speech unit being selected from among a collection of speech units based on the speech unit representation output by the encoder; and
providing, by the one or more computers and as output of the text-to-speech system, audio data for a synthesized utterance of the text that includes the selected speech unit.

11. The system of claim 10 , wherein the encoder is configured to provide speech unit representations of a same size to represent speech units having different durations.

12. The system of claim 10 , wherein the encoder is trained to infer speech unit representations from linguistic unit identifiers, wherein the speech unit representations output by the encoder are vectors that have a same fixed length.

13. The system of claim 10 , wherein the encoder comprises a trained neural network having one or more long-short-term memory layers.

14. One or more non-transitory computer-readable media storing instructions that, when executed by one or more computers, cause the one or more computers to perform operations comprising:
obtaining, by the one or more computers, data indicating a text for text-to-speech synthesis;
providing, by the one or more computers, data indicating a linguistic unit of the text as input to an encoder, the encoder being configured to output speech unit representations indicative of acoustic characteristics based on linguistic information, wherein the encoder is configured to provide speech unit representations learned through machine learning training, wherein the encoder comprises a neural network that was tra

Assignees

• Google Llc

Inventors

• Chun Byung Ha
• Gonzalvo Javier
• Chan Chun-An
• Agiomyrgiannakis Ioannis
• Leung Wan Vincent Ping
• Clark Robert Andrew James
• Vit Jakub

Classifications

• G10L19/0019

  Physics · mapped topic

• G10L13/06

  Elementary speech units used in speech synthesisers; Concatenation rules · CPC title

• G10L13/047 Primary

  Architecture of speech synthesisers · CPC title

• G10L25/30

  using neural networks · CPC title

• G10L13/027

  Concept to speech synthesisers; Generation of natural phrases from machine-based concepts (generation of parameters for speech synthesis out of text G10L13/08) · CPC title