Realistic neural network based image style transfer

Patent metadata

Field	Value
Publication number	US-10891723-B1
Application number	US-201816147705-A
Country	US
Kind code	B1
Filing date	Sep 29, 2018
Priority date	Sep 29, 2017
Publication date	Jan 12, 2021
Grant date	Jan 12, 2021

Abstract

A mobile device can implement a neural network-based style transfer scheme to modify an image in a first style to a second style. The style transfer scheme can be configured to detect an object in the image, apply an effect to the image, and blend the image using color space adjustments and blending schemes to generate a realistic result image. The style transfer scheme can further be configured to efficiently execute on the constrained device by removing operational layers based on resources available on the mobile device.

First claim

What is claimed is:

1. A method comprising:
generating, using an image sensor on a client device, an initial image, the initial image depicting a user face having a first appearance;
receiving an indication of an instruction to modify the initial image, the instruction specifying changing the user face from having the first appearance to a second appearance;
detecting the user face in the initial image;
generating a cropped portion comprising the user face and a non-cropped portion;
generating a modified image from the cropped portion using a convolutional neural network, the convolutional neural network trained on a set of images including images of user faces exhibiting the first appearance and images of user faces exhibiting the second appearance, the modified image displaying the user face having the second appearance;
generating a result image by blending the modified image with the initial image;
applying adjustments to a non-cropped portion of the result image, the non-cropped portion of the result image corresponding to the non-cropped portion of the initial image, and wherein the adjustments are effects selected based on the convolutional neural network used to generate the modified image; and
storing the result image.

2. The method of claim 1 , wherein the convolutional neural network comprises a plurality of downsampling convolution layers that input into a plurality of residual block layers.

3. The method of claim 1 , wherein the result image is generated by blending the modified image with the initial image using Laplacian blending.

4. The method of claim 1 , wherein the modified image is generated not using server-side resources.

5. The method of claim 1 , further comprising:
adjusting color values of the modified image spatially close to the modified image.

6. The method of claim 1 , wherein the result image is generated by blending the modified image into the initial image using Laplacian Pyramid blending.

7. The method of claim 1 , further comprising:
publishing the result image as an ephemeral message on a social network site.

8. The method of claim 1 further comprising:
selecting the convolutional neural network from a plurality of convolution neural networks, wherein the convolution neural network was trained exclusively to change user faces to the second appearance.

9. The method of claim 1 , wherein the second appearance is an aged face and the effects selected is to color hair in the non-cropped portion to appear gray.

10. The method of claim 2 , further comprising:
identifying a model type of the client device; and
decreasing a number of residual blocks implemented in the plurality of residual block layers based on the model type of the client device not satisfying a pre-specified computational resource threshold.

11. The method of claim 2 , wherein the plurality of residual block layers input into a plurality of upsampling convolution layers.

12. The method of claim 5 , wherein adjusting color values comprises:
transferring the modified image from a first color space to a second color space;
transferring the cropped portion from a first color space to a second color space; and
adjusting the second color space of the modified image to colors that are closer to the second color space of the cropped portion.

13. The method of claim 12 , further comprising:
transferring the modified image from the second color space to the first color space.

14. The method of claim 13 , wherein the first color space is a RGB color space and the second color space is a YUV color space.

15. The method of claim 13 , wherein adjustments are implemented using a histogram matching scheme that adjusts a distribution of the second color space of the modified image.

16. A system comprising:
one or more processors of a client device;
an image sensor; and
a memory storing instructions that, when executed by the one or more processors, cause the system to perform operations comprising:
generating, using an image sensor on a client device, an initial image, the initial image depicting a user face having a first appearance;
receiving an indication of an instruction to modify the initial image, the instruction specifying changing the user face from having the first appearance to a second appearance;
detecting the user face in the initial image;
generating a cropped portion comprising the user face and a non-cropped portion;
generating a modified image from the cropped portion using a convolutional neural network, the convolutional neural network trained on a set of images including images of user faces exhibiting the first appearance and images of user faces exhibiting the second appearance, the modified image displaying the user face having the second appearance;
generating a result image by blending the modified image with the initial image;
apply adjustments to a non-cropped portion of the result image, the non-cropped portion of the result image corresponding to the non-cropped portion of the initial image, and wherein the adjustments are effects selected based on the convolutional neural network used to generate the modified image; and
storing the result image.

17. The system of claim 16 , wherein the convolutional neural network comprises a plurality of downsampling convolution layers that input into a plurality of residual block layers.

18. The system of claim 17 , further comprising:
identifying a model type of the client device; and
decreasing a number of residual blocks implemented in the plurality of residual block layers based on the model type of the client device not satisfying a pre-specified computational resource threshold.

19. A non-transitory computer-readable storage medium embodying instructions that, when executed by a device, cause the device to perform operations comprising:
generating, using an image sensor on a client device, an initial image, the initial image depicting a user face having a first appearance;
receiving an indication of an instruction to modify the initial image, the instruction specifying changing the user face from having the first appearance to a second appearance;
detecting the user face in the initial image;
generating a cropped portion comprising the user face and a non-cropped portion;
generating a modified image from the cropped portion using a convolutional neural network, the convolutional neural network trained on a set of images including images of user faces exhibiting the first appearance and images of user faces exhibiting the second appearance, the modified image displaying the user face having the second appearance;
generating a result image by blending the modified image with the initial image;
apply adjustments to a non-cropped portion of the result image, the non-cropped portion of the result image corresponding to the non-cropped portion of the initial image, and wherein the adjustments are effects selected based on the convolutional neural network used to generate the modified image;
and
storing the result image.

20. The non-transitory computer-readable storage medium of claim 19 , wherein the convolutional neural network comprises a plurality of downsampling convolution layers that input into a plurality of residual block layers.

Assignees

• Snap Inc

Inventors

• Chung Jaewook
• Crutchfield Christopher Yale
• Yamangil Emre

Classifications

• H04L67/04 Primary

  specially adapted for terminals or networks with limited capabilities; specially adapted for terminal portability · CPC title

• G06V10/7753

  Incorporation of unlabelled data, e.g. multiple instance learning [MIL] · CPC title

• G06V40/175

  Static expression · CPC title

• G06V10/82

  using neural networks · CPC title

• G06T5/40 Primary

  using histogram techniques · CPC title