Scene graph generation for unlabeled data

Patent metadata

Field	Value
Publication number	US-11574155-B2
Application number	US-202117226561-A
Country	US
Kind code	B2
Filing date	Apr 9, 2021
Priority date	May 27, 2020
Publication date	Feb 7, 2023
Grant date	Feb 7, 2023

Abstract

Approaches are presented for training and using scene graph generators for transfer learning. A scene graph generation technique can decompose a domain gap into individual types of discrepancies, such as may relate to appearance, label, and prediction discrepancies. These discrepancies can be reduced, at least in part, by aligning the corresponding latent and output distributions using one or more gradient reversal layers (GRLs). Label discrepancies can be addressed using self-pseudo-statistics collected from target data. Pseudo statistic-based self-learning and adversarial techniques can be used to manage these discrepancies without the need for costly supervision from a real-world dataset.

First claim

What is claimed is:

1. A computer-implemented method, comprising:
encoding, to a latent space, one or more features of a first set of labeled synthetic data and one or more features of a second set of unlabeled real data;
providing the latent space as an input to train a scene graph prediction network;
aligning one or more features in the latent space and one or more features in an output space of the scene graph prediction network;
aligning one or more labels of the synthetic data with the real data; and
training the scene graph prediction network using the one or more aligned labels.

2. The computer-implemented method of claim 1 , further comprising:
aligning the one or more features in the latent space and the one or more features in the output space using at least one of: one or more gradient reversal layers (GRLs) or a domain discriminator.

3. The computer-implemented method of claim 2 , wherein aligning the one or more features reduces one or more discrepancies in at least one of: an appearance or a content between the first set of labeled synthetic data and the second set of unlabeled real data.

4. The computer-implemented method of claim 1 , further comprising:
aligning the one or more labels using pseudo statistics-based self-learning.

5. The computer-implemented method of claim 1 , further comprising:
receiving an image; and
generating a scene graph for the image using the trained scene graph prediction network.

6. The computer-implemented method of claim 1 , wherein the training comprises applying a network convergence criterion.

7. The computer-implemented method of claim 1 , further comprising:
generating a scene graph using the trained scene graph prediction network; and
generating a synthesized image from the generated scene graph.

8. The computer-implemented method of claim 1 , wherein the one or more features are encoded to the latent space using one or more convolutional neural networks (CNNs), the latent space including one or more features for both the first set of labeled synthetic data and the second set of unlabeled real data.

9. The computer-implemented method of claim 1 , wherein aligning the one or more features reduces an appearance gap between the synthetic data and the real data, and wherein aligning the one or more labels reduces a content gap between the synthetic data and the real data.

10. A system, comprising:
at least one processor; and
memory including instructions that, when executed by the at least one processor, cause the system to:
encode, to a latent space, one or more features of a first set of labeled synthetic data and a second set of unlabeled real data;
provide the latent space as an input to train a scene graph prediction network;
align the one or more features in the latent space and one or more features of an output space of the scene graph prediction network;
align one or more labels between the synthetic data and the real data; and
train the scene graph prediction network using the one or more aligned labels.

11. The system of claim 10 , wherein the instructions when executed further cause the system to:
align the one or more features in the latent space and the one or more features in the output space using at least one of: one or more gradient reversal layers (GRLs) or a domain discriminator.

12. The system of claim 11 , wherein aligning the one or more features reduces one or more discrepancies in appearance and content between the first set of labeled synthetic data and the second set of unlabeled real data.

13. The system of claim 10 , wherein the instructions when executed further cause the system to:
align one or more labels using pseudo statistics-based self-learning.

14. The system of claim 10 , wherein the instructions when executed further cause the system to:
receive an unlabeled image; and
generate a scene graph for the image using the trained scene graph prediction network.

15. The system of claim 10 , wherein the system comprises at least one of:
a system for performing graphical rendering operations;
a system for performing simulation operations;
a system for performing simulation operations to test or validate autonomous machine applications;
a system for performing deep learning operations;
a system implemented using an edge device;
a system incorporating one or more Virtual Machines (VMs);
a system implemented at least partially in a data center; or
a system implemented at least partially using cloud computing resources.

16. A non-transitory computer-readable storage medium including instructions that, when executed, cause one or more processors to:
encode, to a latent space, one or more features of a first set of labeled synthetic data and one or more features of a second set of unlabeled real data;
provide the latent space as an input to train a scene graph prediction network;
align the one or more features in the latent space and the one or more features in the output space of the scene graph prediction network;
align one or more labels of the synthetic data with the real data; and
train the scene graph prediction network using the one or more labels.

17. The non-transitory computer-readable storage medium of claim 16 , wherein the instructions when executed further cause one or more processors to:
align the one or more features in the latent space and the one or more features in the output space using at least one of: gradient reversal layers (GRLs) or a domain discriminator.

18. The non-transitory computer-readable storage medium of claim 17 , wherein aligning the one or more features reduces one or more discrepancies in appearance and content between the first set of labeled synthetic data and the second set of unlabeled real data.

19. The non-transitory computer-readable storage medium of claim 16 , wherein the instructions when executed further cause one or more processors to:
align the one or more labels using pseudo statistics-based self-learning.

20. The non-transitory computer-readable storage medium of claim 16 , wherein the instructions when executed further cause one or more processors to:
receive an unlabeled image; and
generate a scene graph for the image using the trained scene graph prediction network.

Assignees

• Nvidia Corp

Inventors

• Prakash Aayush
• Debnath Shoubhik
• Lafleche Jean-Francois
• Cameracci Eric
• State Gavriel
• Law Marc Teva

Classifications

• G06F18/10

  Pre-processing; Data cleansing · CPC title

• G06V20/00

  Scenes; Scene-specific elements (control of digital cameras H04N23/60) · CPC title

• G06V10/764

  using classification, e.g. of video objects · CPC title

• G06V20/56

  exterior to a vehicle by using sensors mounted on the vehicle · CPC title

• G06F18/29

  Graphical models, e.g. Bayesian networks · CPC title