Image generation using one or more neural networks

Patent metadata

Field	Value
Publication number	US-2022012568-A1
Application number	US-202016922214-A
Country	US
Kind code	A1
Filing date	Jul 7, 2020
Priority date	Jul 7, 2020
Publication date	Jan 13, 2022
Grant date	—

Abstract

Apparatuses, systems, and techniques are presented to generate image or video content. In at least one embodiment, one or more neural networks are used to add one or more first objects to an image including one or more second objects, wherein one or more poses of the one or more first objects in the image is determined with respect to the one or more second objects.

First claim

What is claimed is:

1 . A processor, comprising:
one or more circuits to use one or more neural networks to add one or more first objects to an image including one or more second objects, wherein one or more poses of the one or more first objects in the image is determined with respect to the one or more second objects.

2 . The processor of claim 1 , wherein the one or more neural networks include one or more variational autoencoders (VAEs) to determine features for the first objects and the second objects and encode those features to a latent space to act as a constraint in adding the one or more first objects to the image.

3 . The processor of claim 2 , wherein the one or more neural networks include a gating network to select the one or more VAEs from a set of VAEs each trained for a different class of object, the gating network to select the one or more VAEs using a hierarchical mixture-of-experts approach.

4 . The processor of claim 2 , wherein the one or more neural networks include a generative network to determine one or more potential poses for the one or more first objects based at least in part upon object types of the one or more first objects and with respect to features of the one or more second objects, wherein information for the potential poses is to be encoded into the latent space.

5 . The processor of claim 4 , wherein the one or more neural networks include a neural network to determine one or more potential positions for the one or more first objects based at least in part upon object types and potential poses of the one or more first objects, and with respect to the features of the one or more second objects, wherein information for the potential positions is to be encoded into the latent space.

6 . The processor of claim 4 , wherein the one or more neural networks include a generative adversarial network (GAN) to generate one or more output images including the one or more first objects added to the image, wherein the one or more objects have different poses or positions in the output images, the poses and positions to be selected from the potential poses and the potential positions determined from the latent space.

7 . A system comprising:
one or more processors to use one or more neural networks to add one or more first objects to an image including one or more second objects, wherein one or more poses of the one or more first objects in the image is determined with respect to the one or more second objects.

8 . The system of claim 7 , wherein the one or more neural networks include one or more variational autoencoders (VAEs) to determine features for the first objects and the second objects and encode those features to a latent space to act as a constraint in adding the one or more first objects to the image.

9 . The system of claim 8 , wherein the one or more neural networks include a gating network to select the one or more VAEs from a set of VAEs each trained for a different class of object, the gating network to select the one or more VAEs using a hierarchical mixture-of-experts approach.

10 . The system of claim 8 , wherein the one or more neural networks include a generative network to determine one or more potential poses for the one or more first objects based at least in part upon object types of the one or more first objects and with respect to features of the one or more second objects, wherein information for the potential poses is to be encoded into the latent space.

11 . The system of claim 10 , wherein the one or more neural networks include a neural network to determine one or more potential positions for the one or more first objects based at least in part upon object types and potential poses of the one or more first objects, and with respect to the features of the one or more second objects, wherein information for the potential positions is to be encoded into the latent space.

12 . The system of claim 10 , wherein the one or more neural networks include a generative adversarial network (GAN) to generate one or more output images including the one or more first objects added to the image, wherein the one or more objects have different poses or positions in the output images, the poses and positions to be selected from the potential poses and the potential positions determined from the latent space.

13 . A method comprising:
using one or more neural networks to add one or more first objects to an image including one or more second objects, wherein one or more poses of the one or more first objects in the image is determined with respect to the one or more second objects.

14 . The method of claim 13 , wherein the one or more neural networks include one or more variational autoencoders (VAEs) to determine features for the first objects and the second objects and encode those features to a latent space to act as a constraint in adding the one or more first objects to the image.

15 . The method of claim 14 , wherein the one or more neural networks include a gating network to select the one or more VAEs from a set of VAEs each trained for a different class of object, the gating network to select the one or more VAEs using a hierarchical mixture-of-experts approach.

16 . The method of claim 14 , wherein the one or more neural networks include a generative network to determine one or more potential poses for the one or more first objects based at least in part upon object types of the one or more first objects and with respect to features of the one or more second objects, wherein information for the potential poses is to be encoded into the latent space.

17 . The method of claim 16 , wherein the one or more neural networks include a neural network to determine one or more potential positions for the one or more first objects based at least in part upon object types and potential poses of the one or more first objects, and with respect to the features of the one or more second objects, wherein information for the potential positions is to be encoded into the latent space.

18 . The method of claim 16 , wherein the one or more neural networks include a generative adversarial network (GAN) to generate one or more output images including the one or more first objects added to the image, wherein the one or more objects have different poses or positions in the output images, the poses and positions to be selected from the potential poses and the potential positions determined from the latent space.

19 . A machine-readable medium having stored thereon a set of instructions, which if performed by one or more processors, cause the one or more processors to at least:
use one or more neural networks to generate one or more images indicating one or more interactions between a user and one or more objects in the one or more images.

20 . The machine-readable medium of claim 19 , wherein the one or more neural networks include one or more variational autoencoders (VAEs) to determine features for the first objects and the second objects and encode those features to a latent space to act as a constraint in adding the one or more first objects to the image.

21 . The machine-readable medium of claim 20 , wherein the one or more neural networks include a gating network to select the one or more VAEs from a set of VAEs each trained for a different class of object, the gating network to select the one or more VAEs using a hierarchical mixture-of-experts approach.

22 . The machine-readable medium of claim 20 , wherein the one or more n

Assignees

• Nvidia Corp

Inventors

• Pardeshi Siddhant
• Kothari Pranit P
• Gaikwad Vinayak Vilas

Classifications

• G06T11/60 Primary

  Creating or editing images; Combining images with text · CPC title

• G06V10/774

  Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting · CPC title

• G06V10/82

  using neural networks · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G06F18/214 Primary

  Generating training patterns; Bootstrap methods, e.g. bagging or boosting · CPC title