Voice synthesis method, model training method, device and computer device

Patent metadata

Field	Value
Publication number	US-12014720-B2
Application number	US-202016999989-A
Country	US
Kind code	B2
Filing date	Aug 21, 2020
Priority date	Jul 25, 2018
Publication date	Jun 18, 2024
Grant date	Jun 18, 2024

Abstract

This application relates to a speech synthesis method and apparatus, a model training method and apparatus, and a computer device. The method includes: obtaining to-be-processed linguistic data; encoding the linguistic data, to obtain encoded linguistic data; obtaining an embedded vector for speech feature conversion, the embedded vector being generated according to a residual between synthesized reference speech data and reference speech data that correspond to the same reference linguistic data; and decoding the encoded linguistic data according to the embedded vector, to obtain target synthesized speech data on which the speech feature conversion is performed. The solution provided in this application can prevent quality of a synthesized speech from being affected by a semantic feature in a mel-frequency cepstrum.

First claim

What is claimed is:

1. A speech synthesis method performed at a computer device having one or more processors and memory storing one or more programs to be executed by the one or more processors, the method comprising:
obtaining linguistic data;
encoding the linguistic data, to obtain encoded linguistic data;
obtaining reference linguistic data and corresponding target reference speech data;
encoding the reference linguistic data, to obtain encoded reference linguistic data;
decoding the encoded reference linguistic data, to obtain synthesized reference speech data;
determining a residual between the target reference speech data and the synthesized reference speech data;
obtaining an embedded vector for speech feature conversion, the embedded vector representing a speaking style feature of a target user and being generated according to the residual between the synthesized reference speech data synthesized from the reference linguistic data different from the linguistic data and the target reference speech data that correspond to the same reference linguistic data; and
decoding the encoded linguistic data by performing the speech feature conversion on the encoded linguistic data according to the embedded vector, to obtain target synthesized speech data corresponding to the linguistic data.

2. The method according to claim 1 , further comprising:
processing the residual by using a residual model; and
generating the embedded vector for speech feature conversion according to a result of a forward operation and a result of a backward operation of the residual model.

3. The method according to claim 2 , wherein the generating the embedded vector for speech feature conversion according to a result of a forward operation and a result of a backward operation of the residual model comprises:
obtaining a first vector outputted in the last time step during the forward operation performed by a forward gated recurrent unit (GRU) layer of the residual model;
obtaining a second vector outputted in the first time step during the backward operation performed by a backward GRU layer of the residual model; and
superposing the first vector and the second vector, to obtain the embedded vector for speech feature conversion.

4. The method according to claim 2 , wherein the processing the residual by using a residual model comprises:
processing the residual by using a dense layer, a forward gated recurrent unit (GRU) layer, and a backward GRU layer of the residual model.

5. The method according to claim 2 , wherein the encoded linguistic data is encoded by using a first encoder, the target synthesized speech data is decoded by using a first decoder, the encoded reference linguistic data is encoded by using a second encoder, the synthesized reference speech data is decoded by using a second decoder, and the embedded vector is obtained by using the residual model.

6. The method according to claim 5 , further comprising:
obtaining training linguistic data and corresponding training speech data;
encoding the training linguistic data by using the second encoder, to obtain second encoded training linguistic data;
decoding the second encoded training linguistic data by using the second decoder, to obtain synthesized training speech data;
generating a training embedded vector according to a residual between the synthesized training speech data and the training speech data by using the residual model;
decoding first encoded training linguistic data according to the training embedded vector, to obtain predicted target synthesized speech data on which the speech feature conversion is performed; and
adjusting the second encoder, the second decoder, the residual model, the first encoder, and the first decoder according to a difference between the predicted target synthesized speech data and the training speech data, and continuing to perform training until a training stop condition is satisfied.

7. The method according to claim 1 , wherein the encoded linguistic data is encoded by using a first encoder, the target synthesized speech data is decoded by using a first decoder, and the method further comprises:
obtaining training linguistic data and corresponding target training speech data;
encoding the training linguistic data by using the first encoder, to obtain first encoded training linguistic data;
obtaining a training embedded vector for speech feature conversion, the training embedded vector being generated according to a residual between training speech data synthesized from the training linguistic data and the target training speech data that correspond to the same training linguistic data;
decoding the first encoded training linguistic data according to the training embedded vector by using the first decoder, to obtain predicted target synthesized speech data on which the speech feature conversion is performed; and
adjusting the first encoder and the first decoder according to a difference between the predicted target synthesized speech data and the target training speech data until a training stop condition is satisfied.

8. The method according to claim 1 , wherein the decoding the encoded linguistic data by performing the speech feature conversion on the encoded linguistic data according to the embedded vector, to obtain target synthesized speech data comprises:
splicing the encoded linguistic data and the embedded vector to obtain a spliced vector; and
decoding the spliced vector to obtain the target synthesized speech data on which the speech feature conversion is performed.

9. The method according to claim 1 , further comprising:
determining a speech amplitude spectrum corresponding to the target synthesized speech data;
converting the speech amplitude spectrum into a speech waveform signal in a time domain; and
generating a speech according to the speech waveform signal.

10. A computer device, comprising a memory and a processor, the memory storing a plurality of computer programs, the computer programs, when executed by the processor, causing the computer device to perform a plurality of operations including:
obtaining linguistic data;
encoding the linguistic data, to obtain encoded linguistic data;
obtaining reference linguistic data and corresponding target reference speech data;
encoding the reference linguistic data, to obtain encoded reference linguistic data;
decoding the encoded reference linguistic data, to obtain synthesized reference speech data;
determining a residual between the target reference speech data and the synthesized reference speech data;
obtaining an embedded vector for speech feature conversion, the embedded vector representing a speaking style feature of a target user and being generated according to the residual between the synthesized reference speech data synthesized from reference linguistic data different from the linguistic data and the target reference speech data that correspond to the same reference linguistic data; and
decoding the encoded linguistic data by performing the speech feature conversion on the encoded linguistic data according to the embedded vector, to obtain target synthesized speech data corresponding to the linguistic data.

11. The computer device according to claim 10 , wherein the plurality of operations further comprise:
processing the residual by using a residual model; and
generating the embedded vector for speech feature conversion according to a result of a forward operation and a result of a backward operation of the residual model.

12. The computer device according to claim 11 , wherein the generating the embedded vector for speech feature conversion according to a result of a forwar

Assignees

• Tencent Tech Shenzhen Co Ltd

Inventors

• Wu Xixin
• Wang Mu
• Kang Shiyin
• Su Dan
• Yu Dong

Classifications

• G06N3/09

  Supervised learning · CPC title

• G06N3/0455

  Auto-encoder networks; Encoder-decoder networks · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/0442

  characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU] · CPC title

• G10L19/02

  using spectral analysis, e.g. transform vocoders or subband vocoders · CPC title