Image realism predictor

Patent metadata

Field	Value
Publication number	US-11068746-B2
Application number	US-201816235697-A
Country	US
Kind code	B2
Filing date	Dec 28, 2018
Priority date	Dec 28, 2018
Publication date	Jul 20, 2021
Grant date	Jul 20, 2021

Abstract

A method for predicting the realism of an object within an image includes generating a training image set for a predetermined object type. The training image set comprises one or more training images at least partially generated using a computer. A pixel level training spatial realism map is generated for each training image of the one or more training images. Each training spatial realism map configured to represent a perceptual realism of the corresponding training image. A predictor is trained using the training image set and the corresponding training spatial realism maps. An image of the predetermined object is received. A spatial realism map of the received image is produced using the trained predictor.

First claim

What is claimed is:

1. A method for predicting the realism of an object within an image, comprising:
generating a training image set for a predetermined object type, the training image set comprising one or more training images at least partially generated using a computer;
generating a pixel level training spatial realism map for each training image of the one or more training images, each training spatial realism map configured to represent a perceptual realism of a corresponding training image;
training a predictor using the training image set and corresponding training spatial realism maps;
receiving an image of the predetermined object type; and
producing a spatial realism map of the received image using the trained predictor.

2. The method of claim 1 , wherein generating a training image set for the predetermined object type comprises distorting at least a portion of a natural image, and wherein a corresponding spatial realism map is defined to have a low realism score in the distorted portion of the natural image and high realism score in undistorted portions.

3. The method of claim 2 , wherein distorting at least a portion of the natural image comprises swapping at least the portion of the natural image with a corresponding portion of a computer-generated image to create a combined image.

4. The method of claim 3 , further comprising smoothly blending the natural image and the computer-generated image.

5. The method of claim 1 , wherein generating a training image set and spatial realism map set for the predetermined object type comprises:
presenting, to a user, each training image of the training image set;
receiving an annotation of each training image from the user; and
generating a pixel level spatial realism map based on the received annotation.

6. The method of claim 5 wherein the annotation comprises one or more marked regions in each training image that appear unrealistic to the user.

7. The method of claim 6 wherein the annotation comprises one or more of a bounding polygon, a circle, and an ellipse.

8. The method of claim 1 , wherein generating a training image set comprises generating the training image set using one or more of a deep convolutional generative adversarial network, a self-attention generative adversarial network (SAGAN), and a boundary equilibrium generative adversarial network (BEGAN).

9. The method of claim 1 , wherein the predetermined object type is a human face.

10. The method of claim 1 , wherein the predictor is implemented as a deep convolutional neural network.

11. The method of claim 1 , wherein the predictor is implemented as a U-Net deep neural network.

12. An image realism predictor, comprising:
a processor; and
a memory storing computer program instructions which when executed by the processor cause the processor to perform operations comprising:
generating a training image set for a predetermined object type, the training image set comprising one or more training images at least partially generated using a computer;
generating a pixel level training spatial realism map for each training image of the one or more training images, each training spatial realism map configured to map a perceptual realism of a corresponding training image;
training a predictor using the using the training image set and corresponding training spatial realism maps;
receiving an image of the predetermined object type; and

producing a spatial realism map of the received image using the trained predictor.

13. The image realism predictor of claim 12 , wherein generating a training image for the predetermined object type comprises distorting at least a portion of a natural image, and wherein a corresponding spatial realism map is defined to have a low realism score in the distorted portion of the natural image and high realism score in undistorted portions.

14. The image realism predictor of claim 13 , wherein distorting at least a portion of the natural image comprises swapping at least the portion of the natural image with a corresponding portion of a computer-generated image to create a combined image.

15. The image realism predictor of claim 14 , wherein the processor is configured to blend the natural image and the computer-generated image.

16. The image realism predictor of claim 12 , wherein generating a training image set and spatial realism map set for the predetermined object type comprises:
presenting, to a user, each training image of the training image set;
receiving an annotation of each training image from the user; and
generating a pixel level spatial realism map based on the received annotation.

17. The image realism predictor of claim 16 , wherein the annotation comprises one or more marked regions in each training image that appear unrealistic to the user.

18. The image realism predictor of claim 17 , wherein the annotation comprises one or more of a bounding polygon, a circle, and an ellipse.

19. The image realism predictor of claim 12 , wherein the processor is configured to generate the training image set using one or more of a deep convolutional generative adversarial network, a self-attention generative adversarial network (SAGAN), and a boundary equilibrium generative adversarial network (BEGAN).

Assignees

• Palo Alto Res Ct Inc

Inventors

• Bala Raja
• Shreve Matthew
• Subramanian Jeyasri
• Li Pei

Classifications

• G06V20/70

  Labelling scene content, e.g. deriving syntactic or semantic representations · CPC title

• G06V10/774

  Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting · CPC title

• G06V10/945

  User interactive design; Environments; Toolboxes · CPC title

• G06V20/10

  Terrestrial scenes (scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59) · CPC title

• G06V10/82

  using neural networks · CPC title