Committed information rate variational autoencoders

Patent metadata

Field	Value
Publication number	US-10671889-B2
Application number	US-201916586014-A
Country	US
Kind code	B2
Filing date	Sep 27, 2019
Priority date	Sep 27, 2018
Publication date	Jun 2, 2020
Grant date	Jun 2, 2020

Abstract

A variational autoencoder (VAE) neural network system, comprising an encoder neural network to encode an input data item to define a posterior distribution for a set of latent variables, and a decoder neural network to generate an output data item representing values of a set of latent variables sampled from the posterior distribution. The system is configured for training with an objective function including a term dependent on a difference between the posterior distribution and a prior distribution. The prior and posterior distributions are arranged so that they cannot be matched to one another. The VAE system may be used for compressing and decompressing data.

First claim

What is claimed is:

1. A variational autoencoder neural network system, comprising:
an input to receive an input data item;
an encoder neural network configured to encode the input data item to determine a set of parameters defining a first, posterior distribution of a set of latent variables;
a subsystem to sample from the posterior distribution to determine values of the set of latent variables;
a decoder neural network configured to receive the values of the set of latent variables and to generate an output data item representing the values of the set of latent variables;
wherein the variational autoencoder neural network system is configured for training with an objective function which has a first term dependent upon a difference between the input data item and the output data item and a second term dependent upon a difference between the posterior distribution and a second, prior distribution of the set of latent variables, and wherein a structure of the prior distribution is different to a structure of the posterior distribution such that the posterior distribution cannot be matched to the prior distribution.

2. The variational autoencoder neural network system as claimed in claim 1 wherein the posterior distribution and the prior distribution each comprise a multivariate Gaussian distribution and wherein a variance of the posterior distribution is a factor of α different to a variance of the prior distribution, where α≠1.

3. The variational autoencoder neural network system as claimed in claim 1 wherein the encoder is configured to determine a sequence of sets of parameters defining a sequence of distributions for a sequence of sets of latent variables, one for each of a plurality of time steps.

4. The variational autoencoder neural network system as claimed in claim 3 wherein the prior distribution comprises an autoregressive distribution such that at each time step the prior distribution depends on the prior distribution at a previous time step.

5. The variational autoencoder neural network system as claimed in claim 4 wherein the values of the set of latent variables at a time step t, are defined by a sum of a times the values of the set of latent variables at a previous time step and a noise component, where |α|<1.

6. The variational autoencoder neural network system as claimed in claim 3 , wherein the decoder neural network is an autoregressive neural network configured to generate a sequence of output data item values each conditional upon previously generated output data item values; and further comprising a system to restrict the values of the set of latent variables passed to the decoder at each time step to those which encode information about values in the sequence of output data values yet to be generated.

7. The variational autoencoder neural network system as claimed in claim 3 further comprising an auxiliary neural network configured to learn the sequence of distributions for the sequence of sets of latent variables.

8. A method of training a variational autoencoder neural network system having an encoder neural network configured to encode an input data item to determine a set of parameters defining a first, posterior distribution of a set of latent variables and a decoder neural network configured to receive values of the set of latent variables sampled from the first, posterior distribution and to generate an output data item representing the values of the set of latent variables, comprising:
receiving training data, the training data comprising training data items;
providing each training data item to an input of the variational autoencoder neural network system to generate a corresponding output data item;
determining a gradient of an objective function from a difference between the training data item and the corresponding output data item and from a difference between the posterior distribution and a prior distribution of a set of latent variables; and
backpropagating the gradient through the variational autoencoder neural network system to adjust parameters of the encoder neural network and of the decoder neural network to optimize the objective function.

9. The method as claimed in claim 8 wherein providing each training data item to the input of the variational autoencoder neural network system to generate a corresponding output data item comprises sampling from the posterior distribution to determine sampled values of the set of latent variables; the method further comprising training an auxiliary neural network concurrently with the encoder neural network and decoder neural network using the sampled values of the set of latent variables.

10. One or more non-transitory computer-readable storage media storing instructions that when executed by one or more computers cause the one or more computers to implement a variational autoencoder neural network system, the variational autoencoder neural network system comprising:
an input to receive an input data item;
an encoder neural network configured to encode the input data item to determine a set of parameters defining a first, posterior distribution of a set of latent variables;
a subsystem to sample from the posterior distribution to determine values of the set of latent variables;
a decoder neural network configured to receive the values of the set of latent variables and to generate an output data item representing the values of the set of latent variables;
wherein the variational autoencoder neural network system is configured for training with an objective function which has a first term dependent upon a difference between the input data item and the output data item and a second term dependent upon a difference between the posterior distribution and a second, prior distribution of the set of latent variables, and wherein a structure of the prior distribution is different to a structure of the posterior distribution such that the posterior distribution cannot be matched to the prior distribution.

11. The computer-readable storage media as claimed in claim 10 wherein the posterior distribution and the prior distribution each comprise a multivariate Gaussian distribution and wherein a variance of the posterior distribution is a factor of α different to a variance of the prior distribution, where α≠1.

12. The computer-readable storage media as claimed in claim 10 wherein the encoder is configured to determine a sequence of sets of parameters defining a sequence of distributions for a sequence of sets of latent variables, one for each of a plurality of time steps.

13. The computer-readable storage media as claimed in claim 12 wherein the prior distribution comprises an autoregressive distribution such that at each time step the prior distribution depends on the prior distribution at a previous time step.

14. The computer-readable storage media as claimed in claim 13 wherein the values of the set of latent variables at a time step t, are defined by a sum of a times the values of the set of latent variables at a previous time step and a noise component, where |α|<1.

15. The computer-readable storage media as claimed in claim 13 , wherein the decoder neural network is an autoregressive neural network configured to generate a sequence of output data item values each conditional upon previously generated output data item values; and further comprising a system to restrict the values of the set of latent variables passed to the decoder at each time step to those which encode information about values in the sequence of output data values yet to be generated.

16. The computer-readable storage media as claimed in clai

Assignees

• Deepmind Tech Ltd

Inventors

• Poole Benjamin
• Van Den Oord Aaron Gerard Antonius
• Razavi-Nematollahi Ali
• Vinyals Oriol

Classifications

• G06K9/6257 Primary

  Physics · mapped topic

• G06N7/005

  Physics · mapped topic

• G06N3/084 Primary

  Backpropagation, e.g. using gradient descent · CPC title

• G06N3/0454

  Physics · mapped topic

• G06V10/82

  using neural networks · CPC title