Speech coding using auto-regressive generative neural networks

Patent metadata

Field	Value
Publication number	US-11024321-B2
Application number	US-201816206823-A
Country	US
Kind code	B2
Filing date	Nov 30, 2018
Priority date	Nov 30, 2018
Publication date	Jun 1, 2021
Grant date	Jun 1, 2021

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for coding speech using neural networks. One of the methods includes obtaining a bitstream of parametric coder parameters characterizing spoken speech; generating, from the parametric coder parameters, a conditioning sequence; generating a reconstruction of the spoken speech that includes a respective speech sample at each of a plurality of decoder time steps, comprising, at each decoder time step: processing a current reconstruction sequence using an auto-regressive generative neural network, wherein the auto-regressive generative neural network is configured to process the current reconstruction to compute a score distribution over possible speech sample values, and wherein the processing comprises conditioning the auto-regressive generative neural network on at least a portion of the conditioning sequence; and sampling a speech sample from the possible speech sample values.

First claim

What is claimed is:

1. A method comprising:
processing, at an encoder computer system and using a parametric speech coder, input speech to determine parametric coding parameters characterizing the input speech;
generating, by the encoder computer system and from the parametric coding parameters, a conditioning sequence;
processing, at the encoder computer system, an input speech sequence that comprises a respective observed sample from the input speech at each of the plurality of time steps using an encoder auto-regressive generative neural network to compute a respective probability distribution for each of the plurality of time steps, wherein, for each time step, the auto-regressive generative neural network is conditioned on at least a portion of the conditioning sequence;
determining, at the encoder computer system and from the probability distributions for a first set of time steps of the plurality of time steps, that a decoder auto-regressive generative neural network will not perform poorly in reconstructing the input speech at the time steps in the first set of time steps when conditioned on at least the portion of the conditioning sequence; and
in response,
providing, at the encoder computer system, parametric coding parameters corresponding to the first set of time steps to a decoder computer system for use in reconstructing the input speech at the time steps in the first set of time steps.

2. The method of claim 1 , further comprising:
determining, at the encoder computer system and from the probability distributions for a second set of time steps of the plurality of time steps, that the decoder auto-regressive generative neural network will perform poorly in reconstructing the input speech at the time steps in the second set of time steps when conditioned on at least a portion of the conditioning sequence; and
in response:
entropy coding, at the encoder computer system and using the probability distributions for the second set of time steps, the speech at the time steps in the second set of time steps to generate entropy coded data for the first set of time steps; and
providing, at the encoder computer system, the entropy coded data to the decoder computer system for use in reconstructing the input speech corresponding to the first set of time steps.

3. The method of claim 1 , wherein determining, from the probability distributions for a first set of time steps of the plurality of time steps, that a decoder auto-regressive generative neural network will not perform poorly in reconstructing input speech corresponding to the first set of time steps when conditioned on the conditioning data at the first set of time steps, comprises:
determining that the decoder auto-regressive generative neural network will not perform poorly in reconstructing input speech at a particular time step in the first set of time steps based on the score assigned to the observed sample at the particular time step in the probability distribution for the particular time step.

4. The method of claim 1 , wherein the parametric coding parameters comprise one or more of spectral envelope, pitch, or voicing level.

5. The method of claim 1 , wherein the encoder auto-regressive generative neural network and the decoder auto-regressive generative neural network have the same architecture and the same parameter values.

6. The method of claim 1 , wherein the parametric coding parameters are lower-rate than the conditioning sequence, and wherein generating the conditioning sequence comprises repeating parameters at multiple time steps to extend the bandwidth of the parametric coding parameters.

7. The method of claim 1 , further comprising:
obtaining a bitstream of parametric coder parameters characterizing the input speech, the parameters including the parameters for the first set of time steps;
generating, from the parametric coder parameters, a conditioning sequence;
generating a reconstruction of the first speech that includes a respective speech sample at each of a plurality of decoder time steps, comprising, at each time step in the first set of time steps:
processing a current reconstruction sequence using the decoder auto-regressive generative neural network, wherein the current reconstruction sequence includes the speech samples at each time step preceding the time step, wherein the decoder auto-regressive generative neural network is configured to process the current reconstruction to compute a score distribution over possible speech sample values, and wherein the processing comprises conditioning the decoder auto-regressive generative neural network on at least a portion of the conditioning sequence; and
sampling a speech sample from the possible speech sample values as the speech sample at the time step.

8. The method of claim 7 , wherein the speech samples in the current reconstruction sequence include at least one speech sample that was entropy decoded rather than generated using the decoder neural network.

9. The method of claim 1 , wherein the encoder and decoder auto-regressive generative neural networks are convolutional neural networks.

10. The method of claim 1 , wherein the encoder and decoder auto-regressive generative neural networks are recurrent neural networks.

11. A method comprising:
obtaining a bitstream of parametric coder parameters characterizing spoken speech;
generating, from the parametric coder parameters, a conditioning sequence;
generating a reconstruction of the spoken speech that includes a respective speech sample at each of a plurality of decoder time steps, comprising, at each decoder time step:
processing a current reconstruction sequence using an auto-regressive generative neural network, wherein the current reconstruction sequence includes the speech samples at each time step preceding the decoder time step, and wherein the auto-regressive generative neural network is configured to process the current reconstruction to compute a score distribution over possible speech sample values, and wherein the processing comprises conditioning the auto-regressive generative neural network on at least a portion of the conditioning sequence; and
sampling a speech sample from the possible speech sample values as the speech sample at the decoder time step.

12. The method of claim 11 , wherein the parametric coding parameters comprise one or more of spectral envelope, pitch, or voicing level.

13. The method of claim 11 , wherein the parametric coding parameters are lower-rate than the conditioning sequence, and wherein generating the conditioning sequence comprises repeating parameters at multiple time steps to extend the bandwidth of the parametric coding parameters.

14. The method of claim 1 , wherein the decoder auto-regressive generative neural network is a convolutional neural network.

15. The method of claim 1 , wherein the decoder auto-regressive generative neural network is a recurrent neural network.

16. A method comprising:
processing, at an encoder computer system and using a parametric speech coder, input speech to generate parametric coding parameters characterizing the input speech;
generating, by the encoder computer system and from the parametric coding parameters, a conditioning sequence;
obtaining, from the input speech, a sequence of quantized speech values comprising a respective quantized speech value at each of a plurality of time steps:
entropy coding the quantized speech values, comprising:
processing, at the encoder computer system, the sequence of quantized speech values using an encoder auto-regressive generative neural network to compute a r

Assignees

• Google Llc

Inventors

• Kleijn Willem Bastiaan
• Skoglund Jan K
• Luebs Alejandro
• Lim Sze Chie

Classifications

• G10L19/04

  using predictive techniques · CPC title

• G10L19/00 Primary

  Speech or audio signals analysis-synthesis techniques for redundancy reduction, e.g. in vocoders; Coding or decoding of speech or audio signals, using source filter models or psychoacoustic analysis (in musical instruments G10H) · CPC title

• G10L19/0204 Primary

  using subband decomposition · CPC title

• G10L25/30

  using neural networks · CPC title

• G10L19/02

  using spectral analysis, e.g. transform vocoders or subband vocoders · CPC title