Hyper-structure recurrent neural networks for text-to-speech

Patent metadata

Field	Value
Publication number	US-10127901-B2
Application number	US-201414303969-A
Country	US
Kind code	B2
Filing date	Jun 13, 2014
Priority date	Jun 13, 2014
Publication date	Nov 13, 2018
Grant date	Nov 13, 2018

Abstract

The technology relates to converting text to speech utilizing recurrent neural networks (RNNs). The recurrent neural networks may be implemented as multiple modules for determining properties of the text. In embodiments, a part-of-speech RNN module, letter-to-sound RNN module, a linguistic prosody tagger RNN module, and a context awareness and semantic mining RNN module may all be utilized. The properties from the RNN modules are processed by a hyper-structure RNN module that determine the phonetic properties of the input text based on the outputs of the other RNN modules. The hyper-structure RNN module may generate a generation sequence that is capable of being converting to audible speech by a speech synthesizer. The generation sequence may also be optimized by a global optimization module prior to being synthesized into audible speech.

First claim

The invention claimed is:

1. A method for converting text to speech, the method comprising:
receiving text input into a plurality of first level recurrent neural networks;
determining, by a first recurrent neural network in the plurality of first level recurrent neural networks, one or more properties of the text input from the group consisting of: part-of-speech properties, phonemes, linguistic prosody properties, contextual properties, and semantic properties;
determining, by a second recurrent neural network in the plurality of first level recurrent neural networks, one or more properties of the text input from the group consisting of: part-of-speech properties, phonemes, linguistic prosody properties, contextual properties, and semantic properties, wherein the determined one or properties by the second recurrent neural network is different from the determined one or more properties by the first recurrent neural network;
receiving, by a recurrent neural network in a second level, the determined properties from the first recurrent neural network in the plurality of first level recurrent neural networks and the second recurrent neural network in the plurality of first level recurrent neural networks;
determining by the recurrent neural network in the second level, phonetic properties for the text input based on the properties received from the first recurrent neural network in the plurality of first level recurrent neural networks and the second neural network in the plurality of first level recurrent neural networks, wherein the recurrent neural network in the second level is different from the first recurrent neural network in the plurality of first level recurrent neural networks and the second recurrent neural network in the plurality of first level recurrent neural networks; and
based on the determined phonetic properties, generating a generation sequence for synthetization by an audio synthesizer.

2. The method of claim 1 , wherein the one or more properties received are the part-of-speech properties and phonemes.

3. The method of claim 1 , wherein the one or more properties received are the linguistic prosody properties, the contextual properties, and the semantic properties.

4. The method of claim 1 , wherein the one or more properties received are the phonemes, the contextual properties, and the semantic properties.

5. The method of claim 1 , further comprising optimizing the generation sequence.

6. The method of claim 1 , further comprising synthesizing the generation sequence into audible speech.

7. The method of claim 1 , wherein the one or more properties are received as a dense auxiliary input.

8. The method of claim 1 , wherein the text input and the one or more properties are received as a dense auxiliary input.

9. The method of claim 1 , wherein the recurrent neural network in the second level is a part of a hyper-structure module.

10. The method of claim 1 , wherein the one or more properties are received by a hidden layer and an output layer of the recurrent neural network in the second level.

11. A computer storage device, having computer-executable instructions that, when executed by at least one processor, perform a method for converting text-to-speech, the method comprising:
receiving text input into a plurality of first level recurrent neural networks;
determining, by a first recurrent neural network in the plurality of first level recurrent neural networks, one or more properties of the text input from the group consisting of: part-of-speech properties, phonemes, linguistic prosody properties, contextual properties, and semantic properties;
determining, by a second recurrent neural network in the plurality of first level recurrent neural networks, one or more properties of the text input from the group consisting of: part-of-speech properties, phonemes, linguistic prosody properties, contextual properties, and semantic properties, wherein the determined one or properties by the second recurrent neural network is different from the determined one or more properties by the first recurrent neural network;
receiving, by a recurrent neural network in a second level, the determined properties from the first recurrent neural network in the plurality of first level recurrent neural networks and the second recurrent neural networks in the plurality of first level recurrent neural networks;
determining by the recurrent neural network in the second level, phonetic properties for the text input based on the properties received from the first recurrent neural network in the plurality of first level recurrent neural networks and the second neural network in the plurality of first level recurrent neural networks, wherein the recurrent neural network in the second level is different from the first recurrent neural network in the plurality of first level recurrent neural networks and the second recurrent neural network in the plurality of first level recurrent neural networks; and
based on the determined phonetic properties, generating a generation sequence for synthetization by an audio synthesizer.

12. The computer storage device of claim 11 , wherein the one or more properties received are the part-of-speech properties and phonemes.

13. The computer storage device of claim 11 , wherein the one or more properties received are the phonemes, the contextual properties, and the semantic properties.

14. The computer storage device of claim 11 , wherein the method further comprises optimizing the generation sequence.

15. The computer storage device of claim 11 , wherein the method further comprises synthesizing the generation sequence into audible speech.

16. The computer storage device of claim 11 , wherein the one or more properties are received as a dense auxiliary input.

17. The computer storage device of claim 11 , wherein the text input and the one or more properties are received as a dense auxiliary input.

18. The computer storage device of claim 11 , wherein the recurrent neural network in the second level is a part of a hyper-structure module.

19. The computer storage device of claim 11 , wherein the one or more properties are received by a hidden layer and an output layer of the recurrent neural network in the second level.

20. A system for converting text-to-speech comprising: at least one processor; and
memory encoding computer executable instructions that, when executed by at least one processor, perform a method for converting text to speech, the method comprising:
receiving text input into a plurality of first level recurrent neural networks;

determining, by a first recurrent neural network in the plurality of first level recurrent neural networks, one or more properties of the text input from the group consisting of: part-of-speech properties, phonemes, linguistic prosody properties, contextual properties, and semantic properties;
determining, by a second recurrent neural network in the plurality of first level recurrent neural networks, one or more properties of the text input from the group consisting of: part-of-speech properties, phonemes, linguistic prosody properties, contextual properties, and semantic properties, wherein the determined one or properties by the second recurrent neural network is different from the determined one or more properties by the first recurrent neural network;
receiving, by a recurrent neural network in a second level, the determined properties from the first recurrent neural network in the plurality of first level recurrent neural networks and the second r

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Zhao Pei
• Leung Max
• Yao Kaisheng
• Yan Bo
• Zhao Sheng
• Alleva Fileno A

Classifications

• G06N3/044

  Recurrent networks, e.g. Hopfield networks · CPC title

• G10L13/08 Primary

  Text analysis or generation of parameters for speech synthesis out of text, e.g. grapheme to phoneme translation, prosody generation or stress or intonation determination · CPC title

• G06N3/02

  Neural networks · CPC title

• G10L13/10 Primary

  Prosody rules derived from text; Stress or intonation · CPC title

• G06N3/0445

  Physics · mapped topic