Photorealistic image stylization using a neural network model

Patent metadata

Field	Value
Publication number	US-10872399-B2
Application number	US-201916246375-A
Country	US
Kind code	B2
Filing date	Jan 11, 2019
Priority date	Feb 2, 2018
Publication date	Dec 22, 2020
Grant date	Dec 22, 2020

Abstract

Photorealistic image stylization concerns transferring style of a reference photo to a content photo with the constraint that the stylized photo should remain photorealistic. Examples of styles include seasons (summer, winter, etc.), weather (sunny, rainy, foggy, etc.), lighting (daytime, nighttime, etc.). A photorealistic image stylization process includes a stylization step and a smoothing step. The stylization step transfers the style of the reference photo to the content photo. A photo style transfer neural network model receives a photorealistic content image and a photorealistic style image and generates an intermediate stylized photorealistic image that includes the content of the content image modified according to the style image. A smoothing function receives the intermediate stylized photorealistic image and pixel similarity data and generates the stylized photorealistic image, ensuring spatially consistent stylizations.

First claim

What is claimed is:

1. A computer-implemented method, comprising:
processing a photorealistic content image by a sequence of processing layers within an encoder portion of a photo style transfer neural network model to extract content features from the photorealistic content image, wherein each processing layer includes a max pooling layer that generates location data identifying a location of a maximum value of the content features output by the processing layer;
processing an output of the encoder portion based on style features extracted from a photorealistic style image to produce a transformed feature map; and
processing the transformed feature map by a sequence of decoder processing layers within a decoder portion of the photo style transfer neural network model to produce a stylized photorealistic image including content from the photorealistic content image that is modified according to the photorealistic style image, wherein an unpooling layer within each decoder processing layer uses the location data generated by a corresponding one of the max pooling layers to process the transformed feature map.

2. The computer-implemented method of claim 1 , further comprising processing the stylized photorealistic image according to pixel similarity data for the photorealistic content image to produce a smoothed stylized photorealistic image.

3. The computer-implemented method of claim 2 , wherein the pixel similarity data identifies pixels in the stylized photorealistic image that are consistent in color with adjacent pixels.

4. The computer-implemented method of claim 2 , wherein a quadratic function with a closed-form solution is solved to produce the smoothed stylized photorealistic image.

5. The computer-implemented method of claim 2 , wherein the pixel similarity data comprises pixel affinity values.

6. The computer-implemented method of claim 1 , further comprising processing the photorealistic style image by a second photo style transfer neural network model to produce an improved photorealistic style image, wherein the second photo style transfer neural network model has lower feature representation compared with the photo style transfer neural network model.

7. The computer-implemented method of claim 1 , wherein the photorealistic style image is segmented into a first style region and a second style region and the photo style transfer neural network model processes the first style region and a corresponding first content region of the photorealistic content image to produce a first region of the photorealistic style image.

8. The computer-implemented method of claim 1 , wherein the photorealistic content image is segmented into a first content region and a second content region and the photo style transfer neural network model processes the first content region and a corresponding first style region of the photorealistic style image to produce a first region of the photorealistic style image.

9. A system, comprising a processor configured to implement a photo style transfer neural network model comprising an encoder portion and a decoder portion configured to:
process a photorealistic content image by a sequence of processing layers within the encoder portion to extract content features from the photorealistic content image, wherein each processing layer includes a max pooling layer that generates location data identifying a location of a maximum value of the content features output by the processing layer;
process an output of the encoder portion based on style features extracted from a photorealistic style image to produce a transformed feature map; and
processing the transformed feature map by a sequence of decoder processing layers within the decoder portion to produce a stylized photorealistic image including content from the photorealistic content image that is modified according to the photorealistic style image, wherein an unpooling layer within each decoder processing layer uses the location data generated by a corresponding one of the max pooling layers to process the transformed feature map.

10. The system of claim 9 , further comprising a smoothing function module configured to process the stylized photorealistic image according to pixel similarity data for the photorealistic content image to produce a smoothed stylized photorealistic image.

11. The system of claim 10 , wherein the pixel similarity data identifies pixels in the stylized photorealistic image that are consistent in color with adjacent pixels.

12. The system of claim 10 , wherein a quadratic function with a closed-form solution is solved to produce the smoothed stylized photorealistic image.

13. The system of claim 10 , wherein the pixel similarity data comprises pixel affinity values.

14. The system of claim 9 , further comprising a second photo style transfer neural network model configured to process the photorealistic style image to produce an improved photorealistic style image, wherein the second photo style transfer neural network model has lower feature representation compared with the photo style transfer neural network model.

15. The system of claim 9 , wherein the photorealistic style image is segmented into a first style region and a second style region and the photo style transfer neural network model processes the first style region and a corresponding first content region of the photorealistic content image to produce a first region of the photorealistic style image.

16. The system of claim 9 , wherein the photorealistic content image is segmented into a first content region and a second content region and the photo style transfer neural network model processes the first content region and a corresponding first style region of the photorealistic style image to produce a first region of the photorealistic style image.

17. A non-transitory, computer-readable storage medium storing instructions that, when executed by a processing unit, cause the processing unit to:
process a photorealistic content image by a sequence of processing layers within an encoder portion of a photo style transfer neural network model to extract content features from the photorealistic content image, wherein each processing layer includes a max pooling layer that generates location data identifying a location of a maximum value of the content features output by the processing layer;
process an output of the encoder portion based on style features extracted from a photorealistic style image to produce a transformed feature map; and
processing the transformed feature map by a sequence of decoder processing layers within the decoder portion of the photo style transfer neural network model to produce a stylized photorealistic image including content from the photorealistic content image that is modified according to the photorealistic style image, wherein an unpooling layer within each decoder processing layer uses the location data generated by a corresponding one of the max pooling layers to process the transformed feature map.

Assignees

• Nvidia Corp

Inventors

• Li Yijun
• Liu Ming-Yu
• Yang Ming-Hsuan
• Kautz Jan

Classifications

• G06N3/0455 Primary

  Auto-encoder networks; Encoder-decoder networks · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06T5/60

  using machine learning, e.g. neural networks · CPC title

• G06T5/70

  Denoising; Smoothing · CPC title

• G06T7/11

  Region-based segmentation · CPC title