Learning latent structural relations with segmentation variational autoencoders

Patent metadata

Field	Value
Publication number	US-11816533-B2
Application number	US-202016951158-A
Country	US
Kind code	B2
Filing date	Nov 18, 2020
Priority date	Nov 18, 2020
Publication date	Nov 14, 2023
Grant date	Nov 14, 2023

Abstract

Learning disentangled representations is an important topic in machine learning for a wide range of applications. Disentangled latent variables represent interpretable semantic information and reflect separate factors of variation in data. Although generative models may learn latent representations and generate data samples as well, existing models may ignore the structural information among latent representations. Described in the present disclosure are embodiments to learn disentangled latent structural representations from data using decomposable variational auto-encoders, which simultaneously learn component representations and encode component relationships. Embodiments of a novel structural prior for latent representations are disclosed to capture interactions among different data components. Embodiments are applied to data segmentation and latent relation discovery among different data components. Experiments on several datasets demonstrate the utility of the present model embodiments.

First claim

What is claimed is:

1. A computer-implemented method for learning latent structural relations comprising:
receiving an input comprising one or more components;
generating, using an attention network, a mask corresponding to each of the one or more components, the mask indicating probabilities of at least a subset of elements of the input belonging to each of the one or more components;
encoding, using an encoder, the input and each mask into a latent variable, the latent variable comprises a global latent variable and a local latent variable for a corresponding component;
transforming, using a flow-based model comprising one or more flow functions, one or more global latent variables into one or more transformed global latent variables, each flow function implements a forward transformation for one of the one or more global latent variables to obtain a corresponding transformed global latent variable;
generating an aggregated global latent variable based on the one or more transformed global latent variables;
transforming, using the flow-based model, the aggregated global latent variable into one or more reconstructed global latent variables corresponding to the one or more components;
generating, using a decoder, a pair of a reconstructed mask and a reconstructed component for each of the one or more components using one reconstructed global latent variable and one local latent variable corresponding to each of the one or more components; and
using the reconstructed components and the reconstructed masks corresponding to the one or more components for one or more data processing applications in an inference process, or for constructing one or more losses for training in a training process.

2. The computer-implemented method of claim 1 wherein the input is an image with each of the one or more components representing an object in the image and the elements of the input representing pixels in the image, or a data sample with each of the one or more components representing a portion in the data sample and elements of the input representing data points in the data sample.

3. The computer-implemented method of claim 1 wherein the aggregated global latent variable is an average of the one or more transformed global latent variables corresponding to the one or more components.

4. The computer-implemented method of claim 1 wherein the one or more flow functions are invertible functions.

5. The computer-implemented method of claim 4 wherein each reconstructed global latent variable is obtained by a backward transformation of the aggregated global latent variable using an inverted function of a corresponding flow function.

6. The computer-implemented method of claim 5 wherein the one or more flow function of the flow-base model are enforced, by maximizing an evidence lower bound (ELBO) of the flow-base model, to generate the one or more global latent variables for the one or more components of a same value.

7. The computer-implemented method of claim 6 further comprising:
obtaining, via a backward transformation of a transformed global latent variable for a first component of the one or more components, a prediction of a reconstructed global latent variable value for a second component of the one or more components;
obtaining, via a backward transformation of a transformed global latent variable for the second component, a prediction of a reconstructed global latent variable value for the first component; and
determining, using expected variance of prediction error for the prediction of the reconstructed global latent variable value for the second component and expected variance of prediction error for the prediction of the reconstructed global latent variable value for the first component, a causal direction between the first component and the second component.

8. The computer-implemented method of claim 7 wherein:
in response to the expected variance of prediction error for the prediction of the reconstructed global latent variable value for the second component being larger than the expected variance of prediction error for the prediction of the reconstructed global latent variable value for the first component, determining the first component as a cause and the second component as an effect.

9. The computer-implemented method of claim 1 wherein the mask corresponding to each of the one or more components is generated sequentially based on the input and a scope for each component, the scope for each component is obtained based on a scope and a mask corresponding to a previously processed component.

10. A system for learning latent structural relations comprising:
one or more processors; and
a non-transitory computer-readable medium or media comprising one or more sets of instructions which, when executed by at least one of the one or more processors, causes steps to be performed comprising:
receiving an input comprising one or more components;
yielding, using an attention network, a mask corresponding to each of the one or more components, the mask indicating probabilities of at least a subset of elements of the input belonging to each of the one or more components;
encoding, using an encoder, the input and each mask into a latent variable, the latent variable comprises a global latent variable and a local latent variable for a corresponding component;
transforming, using a flow-based model comprising one or more flow functions, one or more global latent variables for the one or more components into one or more transformed global latent variables, each flow function implements a forward transformation for one of the one or more global latent variables to obtain a corresponding transformed global latent variable;
generating an aggregated global latent variable based on the one or more transformed global latent variables;
transforming, using the flow-based model, the aggregated global latent variable into one or more reconstructed global latent variables corresponding to the one or more components;
generating, using a decoder, a pair of a reconstructed mask and a reconstructed component for each of the one or more components using one reconstructed global latent variable and one local latent variable corresponding to each of the one or more components; and
using the reconstructed components and the reconstructed masks corresponding to the one or more components for one or more data processing applications in an inference process, or for constructing one or more losses for training in a training process.

11. The system of claim 10 wherein the input is an image with each of the one or more components representing an object in the image and elements representing pixels in the image, or a data sample with each of the one or more components representing a portion in the data sample and elements representing data points in the data sample.

12. The system of claim 10 wherein the aggregated global latent variable is an average of the transformed global latent variables corresponding to the one or more components.

13. The system of claim 10 wherein the one or more flow functions are invertible functions.

14. The system of claim 13 wherein each reconstructed global latent variable is obtained by a backward transformation of the aggregated global latent variable using an inverted function of a corresponding flow function.

15. The system of claim 14 wherein the one or more flow function of the flow-base model are enforced, by maximizing an evidence lower bound (ELBO) of the flow-base model, to generate global latent variables for the one or more components of a same value.

16. A non-tra

Assignees

• Baidu Usa Llc

Inventors

• Ren Shaogang
• Fei Hongliang
• Li Dingcheng
• Li Ping

Classifications

• G06N3/0455

  Auto-encoder networks; Encoder-decoder networks · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/0475

  Generative networks · CPC title

• G06N3/0895

  Weakly supervised learning, e.g. semi-supervised or self-supervised learning · CPC title

• G06N5/042 Primary

  Backward inferencing · CPC title