Artificial intelligence-based text-to-speech system and method

Patent metadata

Field	Value
Publication number	US-11244669-B2
Application number	US-201916446833-A
Country	US
Kind code	B2
Filing date	Jun 20, 2019
Priority date	May 18, 2017
Publication date	Feb 8, 2022
Grant date	Feb 8, 2022

Abstract

A technique improves training and speech quality of a text-to-speech (TTS) system having an artificial intelligence, such as a neural network. The TTS system is organized as a front-end subsystem and a back-end subsystem. The front-end subsystem is configured to provide analysis and conversion of text into input vectors, each having at least a base frequency, f0, a phenome duration, and a phoneme sequence that is processed by a signal generation unit of the back-end subsystem. The signal generation unit includes the neural network interacting with a pre-existing knowledgebase of phenomes to generate audible speech from the input vectors. The technique applies an error signal from the neural network to correct imperfections of the pre-existing knowledgebase of phenomes to generate audible speech signals. A back-end training system is configured to train the signal generation unit by applying psychoacoustic principles to improve quality of the generated audible speech signals.

First claim

What is claimed is:

1. A text-to-speech (TTS) training system including one or more processors and one or more memories configured to perform operations comprising:
interacting with data of previously generated speech that was derived from recorded audible speech in a pre-existing knowledgebase of phonemes, wherein the previously generated speech has speech signal distortions;
generating a corrected speech signal of the previously generated speech to correct for the speech signal distortions of the pre-existing knowledgebase of phonemes by iteratively training a subsystem for the speech signal distortions of the pre-existing knowledgebase of phonemes based on psychoacoustic processing of the data of the previously generated speech in the pre-existing knowledgebase of phonemes; and
applying the corrected speech signal from the neural network to the previously generated speech for correcting the speech signal distortions of the previously generated speech that was derived from the recorded audible speech in the pre-existing knowledgebase of phonemes.

2. The TTS training system of claim 1 wherein the operations further comprise calculating audible portions of the corrected speech signal by the psychoacoustic processing.

3. The TTS training system of claim 1 wherein the psychoacoustic processing ignores processing of inaudible portions of the corrected speech signal.

4. The TTS training system of claim 1 wherein the operations further comprise iteratively modifying a neural network to correct the subsystem.

5. The TTS training system of claim 1 wherein the operations further comprise analyzing a reference audio signal to determine masking information.

6. The TTS training system of claim 1 wherein the operations further comprise identifying locations and energy levels that are audible and inaudible.

7. The TTS training system of claim 1 wherein the operations further comprise determining a quality indicator based on audible portions of the corrected speech signal and using the psychoacoustic processing to ignore the inaudible portions of the corrected speech signal.

8. The TTS training system of claim 1 wherein the operations further comprise calculating a quality indicator based on a total of audible signal energy.

9. The TTS training system of claim 8 wherein the iterative modification comprises a modification of a neural network that is performed so that the total audible signal energy is below a quality threshold.

10. The TTS training system of claim 1 wherein the operations further comprise outputting audible portions of a phoneme sequence based on masking information.

11. A method of training text-to-speech (TTS) processing comprising:
interacting with data of previously generated speech that was derived from recorded audible speech in a pre-existing knowledgebase of phonemes, wherein the previously generated speech has speech signal distortions;
generating a corrected speech signal of the previously generated speech to correct for the speech signal distortions of the pre-existing knowledgebase of phonemes by iteratively training a subsystem for the speech signal distortions of the pre-existing knowledgebase of phonemes based on psychoacoustic processing of the data of the previously generated speech in the pre-existing knowledgebase of phonemes; and
applying the corrected speech signal from the neural network to the previously generated speech for correcting the speech signal distortions of the previously generated speech that was derived from the recorded audible speech in the pre-existing knowledgebase of phonemes.

12. The method of claim 11 further comprising calculating audible portions of the corrected speech signal by the psychoacoustic processing.

13. The method of claim 11 wherein the psychoacoustic processing ignores processing of inaudible portions of the corrected speech signal.

14. The method of claim 11 wherein the iterative modification comprises modifying a neural network to correct the subsystem.

15. The method of claim 11 further comprising analyzing a reference audio signal to determine masking information.

16. The method of claim 11 further comprising identifying locations and energy levels that are audible and inaudible.

17. The method of claim 11 further comprising:
determining a quality indicator based on audible portions of the corrected speech signal; and
ignoring the inaudible portions of the corrected speech signal using the psychoacoustic processing.

18. The method of claim 11 further comprising calculating a quality indicator based on a total of audible signal energy.

19. The method of claim 18 wherein the iterative modification comprises modifying a neural network so that the total audible signal energy is below a quality threshold.

20. A non-transitory computer-readable medium having program instructions for training text-to-speech (TTS) processing which, when executed across one or more processors, causes at least a portion of the one or more processors to perform operations comprising:
interacting with data of previously generated speech that was derived from recorded audible speech in a pre-existing knowledgebase of phonemes, wherein the previously generated speech has speech signal distortions;
generating a corrected speech signal of the previously generated speech to correct for the speech signal distortions of the pre-existing knowledgebase of phonemes by iteratively training a subsystem for the speech signal distortions of the pre-existing knowledgebase of phonemes based on psychoacoustic processing of the data of the previously generated speech in the pre-existing knowledgebase of phonemes; and
applying the corrected speech signal from the neural network to the previously generated speech for correcting the speech signal distortions of the previously generated speech that was derived from the recorded audible speech in the pre-existing knowledgebase of phonemes.

Assignees

• Telepathy Labs Inc

Inventors

• Reber Martin
• Avijeet Vijeta

Classifications

• G06N3/042

  Knowledge-based neural networks; Logical representations of neural networks · CPC title

• G06F18/217

  Validation; Performance evaluation; Active pattern learning techniques · CPC title

• G06F18/2135

  based on approximation criteria, e.g. principal component analysis · CPC title

• G06N3/08

  Learning methods · CPC title

• G10L25/30

  using neural networks · CPC title