Inaudible watermark enabled text-to-speech framework

Patent metadata

Field	Value
Publication number	US-11138964-B2
Application number	US-201916659550-A
Country	US
Kind code	B2
Filing date	Oct 21, 2019
Priority date	Oct 21, 2019
Publication date	Oct 5, 2021
Grant date	Oct 5, 2021

Abstract

According to various embodiments, an end-to-end TTS framework can integrate a watermarking process into the training of the TTS framework, which enables watermarks to be imperceptible within a synthesized/cloned audio segment generated by the TTS framework. The watermarks added in such a matter are statistically undetectable to prevent authorized removal. According to an exemplary method of training the TTS framework, a TTS neural network model and a watermarking neural network mode in the TTS framework are trained in an end to end manner, with the watermarking being part of the optimization process of the TTS framework. During the training, neuron values of the TTS neural network model are adjusted based on training data to prepare one or more spaces for adding a watermark in a synthesized audio segment to be generated by the TTS framework.

First claim

What is claimed is:

1. A computer-implemented method of training a text to speech (TTS) framework, the method comprising:
receiving, at a TTS framework, a set of training data for training the TTS framework to generate synthesized audio segments with a watermark, wherein the TTS framework includes a TTS neural network model and a watermarking neural network model;
adjusting neuron values of the TTS neural network model to prepare one or more spaces in a synthesized audio segment to be generated by the TTS framework for adding the watermark; and
adjusting neuron values of the watermarking neural network model to add the watermark to the one or more prepared spaces.

2. The method of claim 1 , wherein the TTS framework is trained using the set of training data end to end, including training the TTS neural network model and the watermarking neural network model together.

3. The method of claim 1 , wherein the watermarking neural network model is an invertible neural network that provides a one-to-one mapping between an input audio segment and a watermarked audio segment.

4. The method of claim 1 , wherein the neuron values in each of the TTS neural network model and the watermarking neural network model include weights, biases and activation functions.

5. The method of claim 4 , wherein the neuron values of the TTS neural network model are adjusted during the training of the TTS framework such that the watermark added to the one or more spaces is inaudible in the synthesized audio segment generated by the TTS framework.

6. The method of claim 5 , wherein adding the watermark is performed by a plurality of layers of neurons associated with weights, biases and activation functions in the watermarking neural network model.

7. The method of claim 1 , wherein the TTS framework is trained to generate the synthesized audio segment including one or more speech phrases that are overlapped with a speech phrase representing the watermark, such that the one or more speech phrases cover the watermark speech phrase.

8. The method of claim 7 , wherein one or more physical properties associated with the one or more speech phrases are modified during the training of the TTS framework to cover the watermark speech phrase.

9. The method of claim 8 , wherein modifying the physical properties of the one or more speech phrases includes modifying a length of each of the one or more speech phrases such that each speech phrase covers the watermark phrase.

10. A non-transitory machine-readable medium having instructions stored therein for training a text to speech (TTS) framework, which instructions, when executed by a processor, cause the processor to perform operations, the operations comprising:
receiving, at a TTS framework, a set of training data for training the TTS framework to generate synthesized audio segments with a watermark, wherein the TTS framework includes a TTS neural network model and a watermarking neural network model;
adjusting neuron values of the TTS neural network model to prepare one or more spaces in a synthesized audio segment to be generated by the TTS framework for adding the watermark; and
adjusting neuron values of the watermarking neural network model to add the watermark to the one or more prepared spaces.

11. The non-transitory machine-readable medium of claim 10 , wherein the TTS framework is trained using the set of training data end to end, including training the TTS neural network model and the watermarking neural network model together.

12. The non-transitory machine-readable medium of claim 10 , wherein the watermarking neural network model is an invertible neural network that provides a one-to-one mapping between an input audio segment and a watermarked audio segment.

13. The non-transitory machine-readable medium of claim 10 , wherein the neuron values in each of the TTS neural network model and the watermarking neural network model include weights, biases and activation functions.

14. The non-transitory machine-readable medium of claim 13 , wherein the neuron values of the TTS neural network model are adjusted during the training of the TTS framework such that the watermark added to the one or more spaces is inaudible in the synthesized audio segment generated by the TTS framework.

15. The non-transitory machine-readable medium of claim 14 , wherein adding the watermark is performed by a plurality of layers of neurons associated with weights, biases and activation functions in the watermarking neural network model.

16. The non-transitory machine-readable medium of claim 10 , wherein the TTS framework is trained to generate the synthesized audio segment including one or more speech phrases that are overlapped with a speech phrase representing the watermark, such that the one or more speech phrases cover the watermark speech phrase.

17. The non-transitory machine-readable medium of claim 16 , wherein one or more physical properties associated with the one or more speech phrases are modified during the training of the TTS framework to cover the watermark speech phrase.

18. The non-transitory machine-readable medium of claim 17 , wherein modifying the physical properties of the one or more speech phrases includes modifying a length of each of the one or more speech phrases such that each speech phrase covers the watermark phrase.

19. A data processing system, comprising:
a processor; and
a memory coupled to the processor to store instructions, which when executed by the processor, cause the processor to perform operations, the operations including
receiving, at a TTS framework, a set of training data for training the TTS framework to generate synthesized audio segments with a watermark, wherein the TTS framework includes a TTS neural network model and a watermarking neural network model;
adjusting neuron values of the TTS neural network model to prepare one or more spaces in a synthesized audio segment to be generated by the TTS framework for adding the watermark; and
adjusting neuron values of the watermarking neural network model to add the watermark to the one or more prepared spaces.

20. The system of claim 19 , wherein the watermarking neural network model is an invertible neural network that provides a one-to-one mapping between an input audio segment and a watermarked audio segment.

Assignees

• Baidu Usa Llc

Inventors

• Ping Wei
• Zhong Zhenyu
• Cheng Yueqiang
• Li Xing
• Wei Tao

Classifications

• G10L13/08 Primary

  Text analysis or generation of parameters for speech synthesis out of text, e.g. grapheme to phoneme translation, prosody generation or stress or intonation determination · CPC title

• G10L25/30

  using neural networks · CPC title

• G10L19/018 Primary

  Audio watermarking, i.e. embedding inaudible data in the audio signal · CPC title

• G10L13/047 Primary

  Architecture of speech synthesisers · CPC title