Creating an image utilizing a map representing different classes of pixels

Patent metadata

Field	Value
Publication number	US-2022012536-A1
Application number	US-202117483688-A
Country	US
Kind code	A1
Filing date	Sep 23, 2021
Priority date	Nov 15, 2017
Publication date	Jan 13, 2022
Grant date	—

Abstract

A method, computer readable medium, and system are disclosed for creating an image utilizing a map representing different classes of specific pixels within a scene. One or more computing systems use the map to create a preliminary image. This preliminary image is then compared to an original image that was used to create the map. A determination is made whether the preliminary image matches the original image, and results of the determination are used to adjust the computing systems that created the preliminary image, which improves a performance of such computing systems. The adjusted computing systems are then used to create images based on different input maps representing various object classes of specific pixels within a scene.

First claim

What is claimed is:

1 . A processor, comprising:
one or more circuits to train one or more neural networks to generate an image based, at least in part, on one or more labels of one or more pixels.

2 . The processor of claim 1 , wherein the one or more circuits are further to:
generate a first image using a first neural network of the one or more neural networks; generate a second image using a second neural network of the one or more neural networks based at least in part on the first image; and train the one or more neural networks based on differences between the image and the second image.

3 . The processor of claim 2 , wherein the one or more circuits are further to train the one or more neural networks by at least adjusting one or more values of one or more nodes of the first neural network and the second neural network based at least in part on the differences between the image and the second image.

4 . The processor of claim 1 , wherein the one or more labels indicate one or more classes of objects that the one or more pixels correspond to.

5 . The processor of claim 1 , wherein the one or more neural networks include one or more generator neural networks.

6 . The processor of claim 1 , wherein the one or more circuits are further to train the one or more neural networks using at least one or more feature matching loss functions.

7 . The processor of claim 1 , wherein the one or more labels of the one or more pixels are represented through a semantic label map.

8 . A system, comprising:
one or more computers having one or more processors to train one or more neural networks to generate an image based, at least in part, on one or more labels of one or more pixels.

9 . The system of claim 8 , wherein the one or more processors are further to:
use the one or more neural networks to generate a first representation of the image; generate a second representation of the image based at least in part on the first representation of the image and the one or more labels; and train the one or more neural networks based on the image and the second representation of the image.

10 . The system of claim 9 , wherein a resolution of the second representation of the image is greater than a resolution of the first representation of the image.

11 . The system of claim 9 , wherein the one or more processors are further to train the one or more neural networks by at least using a discriminator neural network to process the image and the second representation of the image.

12 . The system of claim 8 , wherein the one or more processors are further to generate the image using one or more features.

13 . The system of claim 12 , wherein the one or more features indicate one or more properties of the generated image.

14 . A processor, comprising:
one or more circuits to use one or more neural networks to generate an image based, at least in part, on one or more labels of one or more pixels.

15 . The processor of claim 14 , wherein the one or more circuits are further to:
use a first generator to generate a first image based at least in part on the one or more labels of the one or more pixels; and use a second generator to generate the image based at least in part on the first image, wherein the image depicts one or more objects corresponding to the one or more labels of the one or more pixels.

16 . The processor of claim 14 , wherein the one or more circuits are further to:
obtain a set of features, wherein the set of features indicate one or more characteristics of the image; and cause the one or more neural networks to process the set of features to generate the image.

17 . The processor of claim 16 , wherein the one or more circuits are further to generate the set of features using one or more encoders.

18 . The processor of claim 14 , wherein the one or more labels of the one or more pixels are indicated through one or more semantic representations.

19 . The processor of claim 14 , wherein the one or more circuits are further to input the image to one or more systems of an autonomous vehicle for object detection.

20 . A system, comprising:
one or more computers having one or more processors to use one or more neural networks to generate an image based, at least in part, on one or more labels of one or more pixels.

21 . The system of claim 20 , wherein the one or more processors are further to:
cause a first neural network of the one or more neural networks to generate a first representation of the image; and cause a second neural network of the one or more neural networks to generate the image based at least in part on the first representation of the image and the one or more labels of the one or more pixels.

22 . The system of claim 20 , wherein the one or more processors are further to:
obtain one or more features corresponding to one or more objects; and generate the image depicting the one or more objects based at least in part on the one or more features.

23 . The system of claim 22 , wherein the one or more processors are further to obtain the one or more features using one or more pooling operations.

24 . The system of claim 20 , wherein the one or more processors are further to input the image to one or more systems of an autonomous vehicle for one or more navigation tasks.

25 . The system of claim 21 , wherein the first neural network and the second neural network are part of one or more generative adversarial networks.

26 . A machine-readable medium having stored thereon a set of instructions, which if performed by one or more processors, cause the one or more processors to at least:
train one or more neural networks to generate an image based, at least in part, on one or more labels of one or more pixels.

27 . The machine-readable medium of claim 26 , wherein the set of instructions further comprise instructions, which if performed by the one or more processors, cause the one or more processors to:
generate a first digital representation of the image based on the one or more labels of the one or more pixels; generate a second digital representation of the image based on the first digital representation of the image; and train the one or more neural networks by at least comparing the second digital representation of the image with the image.

28 . The machine-readable medium of claim 27 , wherein the set of instructions further comprise instructions, which if performed by the one or more processors, cause the one or more processors to:
process the image to generate a downsampled image; process the second digital representation of the image to generate a downsampled second digital representation of the image; and train the one or more neural networks by at least comparing the downsampled second digital representation of the image with the downsampled image.

29 . The machine-readable medium of claim 26 , wherein the set of instructions further comprise instructions, which if performed by the one or more processors, cause the one or more processors to train the one or more neural networks using one or more discriminators part of one or more generative adversarial networks.

30 . The machine-readable medium of claim 29 , wherein the one or more discriminators include one or more patch-based discriminators.

31 .

Assignees

• Nvidia Corp

Inventors

• Wang Ting-Chun
• Liu Ming-Yu
• Catanzaro Bryan Christopher
• Kautz Jan
• Tao Andrew J

Classifications

• G06V10/454 Primary

  Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN] · CPC title

• G06V20/70

  Labelling scene content, e.g. deriving syntactic or semantic representations · CPC title

• G06V30/19173

  Classification techniques · CPC title

• G06V20/10

  Terrestrial scenes (scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59) · CPC title

• G06V10/82

  using neural networks · CPC title