Domain stylization using a neural network model

Patent metadata

Field	Value
Publication number	US-10984286-B2
Application number	US-201916265725-A
Country	US
Kind code	B2
Filing date	Feb 1, 2019
Priority date	Feb 2, 2018
Publication date	Apr 20, 2021
Grant date	Apr 20, 2021

Abstract

A style transfer neural network may be used to generate stylized synthetic images, where real images provide the style (e.g., seasons, weather, lighting) for transfer to synthetic images. The stylized synthetic images may then be used to train a recognition neural network. In turn, the trained neural network may be used to predict semantic labels for the real images, providing recognition data for the real images. Finally, the real training dataset (real images and predicted recognition data) and the synthetic training dataset are used by the style transfer neural network to generate stylized synthetic images. The training of the neural network, prediction of recognition data for the real images, and stylizing of the synthetic images may be repeated for a number of iterations. The stylization operation more closely aligns a covariate of the synthetic images to the covariate of the real images, improving accuracy of the recognition neural network.

First claim

What is claimed is:

1. A computer-implemented method, comprising:
processing a photorealistic image by a first neural network model to produce predicted recognition data for the photorealistic image, wherein the predicted recognition data comprises a segmentation map that indicates, for each pixel of the photorealistic image covered by an object, a label corresponding to the object, the segmentation map identifies at least two style regions in the photorealistic image, and each style region in the at least two style regions is associated with a different label; and
processing the photorealistic image, the predicted recognition data for the photorealistic image, a synthetic image, and ground truth recognition data for the synthetic image by a second neural network model to produce a stylized synthetic image including content from the synthetic image that is modified according to a style of the photorealistic image.

2. The computer-implemented method of claim 1 , further comprising training the first neural network model using the stylized synthetic image and the ground truth recognition data for the synthetic image.

3. The computer-implemented method of claim 2 , further comprising, after the training, processing the photorealistic image by the first neural network model to produce second predicted recognition data for the photorealistic image.

4. The computer-implemented method of claim 3 , further comprising processing the photorealistic image, the second predicted recognition data, the synthetic image, and the ground truth recognition data by the second neural network model to produce a second stylized synthetic image including content from the synthetic image that is modified according to the style of the photorealistic image.

5. The computer-implemented method of claim 1 , further comprising processing additional photorealistic images, predicted recognition data for the additional photorealistic images, the synthetic image, and the ground truth recognition data for the synthetic image by the second neural network model to produce additional stylized synthetic images, each including content from the synthetic image that is modified according a different style of each of the additional photorealistic images.

6. The computer-implemented method of claim 1 , prior to processing the photorealistic image by the first neural network model, processing a set of photorealistic images without recognition data for the set of photorealistic images, a set of synthetic images, and ground truth recognition data for the set of synthetic images by the second neural network model to produce a set of stylized synthetic images.

7. The computer-implemented method of claim 1 , wherein the ground truth recognition data for the synthetic image identifies a first region associated with a first label, and a first region of the stylized synthetic image is stylized based on a first region of the photorealistic image that, according to the predicted recognition data for the photorealistic image, is associated with the first label.

8. The computer-implemented method of claim 7 , wherein the ground truth recognition data for the synthetic image identifies a second region associated with a second label, and a second region of the stylized synthetic image is unchanged compared to the second region in the synthetic image responsive to determining that the predicted recognition data for the photorealistic image does not identify a region of the photorealistic image that is associated with the second label.

9. The computer-implemented method of claim 1 , wherein the first neural network model is a classification neural network model.

10. The computer-implemented method of claim 1 , wherein the first neural network model is a segmentation neural network model.

11. The computer-implemented method of claim 1 , wherein the first neural network model is an object detection neural network model.

12. The computer-implemented method of claim 1 , wherein the first neural network model is an instance segmentation neural network model.

13. The computer-implemented method of claim 1 , wherein the first neural network model is a panoptic segmentation neural network model.

14. A computer-implemented method, comprising:
training a first neural network model using stylized synthetic images and ground truth recognition data for synthetic images, wherein the stylized synthetic images include content from the synthetic images that is modified according to styles of photorealistic images by a second neural network model; and
processing a particular photorealistic image by the first neural network model to produce predicted recognition data for the photorealistic image, wherein the predicted recognition data comprises a segmentation map that indicates, for each pixel of the particular photorealistic image covered by an object, a label corresponding to the object, the segmentation map identifies at least two style regions in the particular photorealistic image, and each style region in the at least two style regions is associated with a different label.

15. The computer-implemented method of claim 14 , wherein the particular photorealistic image is included as one of the photorealistic images processed by the second neural network model to produce a corresponding stylized synthetic image.

16. A system, comprising:
a first neural network model implemented by at least one processor and configured to process a photorealistic image to produce predicted recognition data for the photorealistic image, wherein the predicted recognition data comprises a segmentation map that indicates, for each pixel of the photorealistic image covered by an object, a label corresponding to the object, wherein the segmentation map identifies at least two style regions in the photorealistic image, and each style region in the at least two style regions is associated with a different label; and
a second neural network model implemented by the at least one processor and configured to process the photorealistic image, the predicted recognition data for the photorealistic image, a synthetic image, and ground truth recognition data for the synthetic image to produce a stylized synthetic image including content from the synthetic image that is modified according to a style of the photorealistic image.

17. The system of claim 16 , wherein the first neural network model is trained using the stylized synthetic image and the ground truth recognition data for the synthetic image.

18. The system of claim 17 , wherein, after the training, the first neural network model processes the photorealistic image to produce second predicted recognition data for the photorealistic image.

19. The system of claim 18 , wherein the second neural network model processes the photorealistic image, the second predicted recognition data, the synthetic image, and the ground truth recognition data to produce a second stylized synthetic image including content from the synthetic image that is modified according to a style of the photorealistic image.

20. A non-transitory, computer-readable storage medium storing instructions that, when executed by a processing unit, cause the processing unit to:
process a photorealistic image by a first neural network model to produce predicted recognition data for the photorealistic image, wherein the predicted recognition data comprises a segmentation map that indicates, for each pixel of the photorealistic image covered by an object, a label corresponding to the object, the segmentation map identifies at least

Assignees

• Nvidia Corp

Inventors

• Dundar Aysegul
• Liu Ming-Yu
• Wang Ting-Chun
• Zedlewski John
• Kautz Jan

Classifications

• G06V10/774

  Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting · CPC title

• G06V10/82 Primary

  using neural networks · CPC title

• G06V10/764

  using classification, e.g. of video objects · CPC title

• G06F18/214

  Generating training patterns; Bootstrap methods, e.g. bagging or boosting · CPC title

• G06F18/24133

  Distances to prototypes · CPC title