Generating a stylized image or stylized animation by matching semantic features via an appearance guide, a segmentation guide, and/or a temporal guide

Patent metadata

Field	Value
Publication number	US-10504267-B2
Application number	US-201715784415-A
Country	US
Kind code	B2
Filing date	Oct 16, 2017
Priority date	Jun 6, 2017
Publication date	Dec 10, 2019
Grant date	Dec 10, 2019

Abstract

Certain embodiments involve generating an appearance guide, a segmentation guide, and a positional guide and using one or more of the guides to synthesize a stylized image or animation. For example, a system obtains data indicating a target and a style exemplar image and generates a segmentation guide for segmenting the target image and the style exemplar image and identifying a feature of the target image and a corresponding feature of the style exemplar image. The system generates a positional guide for determining positions of the target feature and style feature relative to a common grid system. The system generates an appearance guide for modifying intensity levels and contrast values in the target image based on the style exemplar image. The system uses one or more of the guides to transfer a texture of the style feature to the corresponding target feature.

First claim

What is claimed is:

1. A method for generating a stylized image, the method comprising:
obtaining, by a processor, data indicating a target image and a style exemplar image, the target image including a first character and the style exemplar image including a second character and wherein the style exemplar image includes a style or texture;
generating, by the processor, a segmentation guide for the target image and the style exemplar image by:
creating a first head soft mask of the first character and a second head soft mask of the second character;
identifying a plurality of target features of the target image and a plurality of style features of the style exemplar image; and
creating a soft mask for a target feature of the plurality of target features and a soft mask for a style feature of the plurality of style features;

creating, by the processor and using the segmentation guide, the stylized image by applying the texture or style from the style exemplar image to the target image; and
outputting, by the processor, the stylized image for display via a user interface or for storage.

2. The method of claim 1 , wherein the first character is the same as the second character.

3. The method of claim 1 , further comprising:
generating, by the processor, a positional guide for the target image and the style exemplar image by:
identifying a correspondence between a first position of the target feature or another target feature of the plurality of target features and a second position of the style feature or another style feature of the plurality of style features relative to a common grid; and
warping the second position of the style feature to the first position of the target feature, and wherein creating the stylized image further comprises:
applying, by the processor and using the segmentation guide and the positional guide, a first texture or style of the style exemplar image at the second position of the style feature to the target image at the first position of the target feature.

4. The method of claim 3 , further comprising:
generating, by the processor, an appearance guide for the style exemplar image and the target image by:
generating a converted target image and a converted style exemplar image by converting a copy of the style exemplar image and a copy of the target image into grayscale;
matching a global intensity level of the converted target image to a global intensity level of the converted style exemplar image;
matching a local and global contrast of the converted target image to a local and global contrast of the converted style exemplar image;
identifying a set of pixels in the converted target image having an intensity level that matches an intensity level of a set of pixels in the converted style exemplar image; and
transferring the style or texture of the style exemplar image to the target image using the identified set of pixels in the converted style exemplar image.

5. The method of claim 1 , further comprising:
generating, by the processor, an appearance guide for the style exemplar image and the target image by:
generating a converted target image and a converted style exemplar image by converting a copy of the style exemplar image and a copy of the target image into grayscale;
matching a global intensity level of the converted target image to a global intensity level of the converted style exemplar image;
matching a local and global contrast of the converted target image to a local and global contrast of the converted style exemplar image;
identifying a set of pixels in the converted target image having an intensity level that matches an intensity level of a set of pixels in the converted style exemplar image; and
transferring the style or texture of the style exemplar image to the target image using the identified set of pixels in the converted style exemplar image.

6. The method of claim 1 , further comprising:
generating, by the processor, a temporal guide for generating a stylized video, the stylized video comprising a plurality of stylized images and the temporal guide usable for determining an amount of temporal flickering in the stylized video, wherein generating the temporal guide comprises advecting, by the processor and using an underlying motion field, a stylized image of the plurality of stylized images;
controlling, by the processor, the amount of temporal flickering in the stylized video by varying an amount of blur of the advected stylized image and the style exemplar image; and
outputting, by the processor, the stylized video for display via the user interface or for storage.

7. The method of claim 1 , wherein generating the segmentation guide further comprises creating a first hair region soft mask of the first character and a second hair region soft mask of the second character, wherein creating the first head soft mask of the first character and the second head soft mask of the second character comprises:
removing, by the processor, a foreground mask from the target image to create a trimap of the first character and a foreground mask from the style exemplar image to create a trimap of the second character;
identifying, by the processor, a landmark target feature of the first character and refining the trimap of the first character using the landmark target feature;
identifying, by the processor, a landmark style feature of the second character and refining the trimap of the second character using the landmark style feature; and
generating, by the processor and using a matting algorithm, the first head soft mask based on the refined trimap of the first character and the second head soft mask based on the refined trimap of the second character.

8. The method of claim 1 , wherein generating the segmentation guide further comprises creating a first hair region soft mask of the first character and a second hair region soft mask of the second character, wherein creating the first hair region soft mask of the first character and the second hair region soft mask of the second character comprises:
generating, by the processor, a first skin region trimap of the first character and a second skin region trimap of the second character by:
converting the target image and the style exemplar image to YC B C R color space; and
determining a likelihood of each pixel in the target image and each pixel in the style exemplar image being a skin pixel by using a color classification algorithm, wherein determining the likelihood of a pixel being a skin pixel comprises comparing the pixel to a threshold value;

creating a first skin region soft mask based on the first skin region trimap and a second skin region soft mask based on the second skin region trimap;
segmenting the first character into the first hair region soft mask and the first skin region soft mask based on a first pixelwise difference between the first head soft mask and the first skin region soft mask, wherein a pixelwise difference corresponds to a difference between pixels in the first head soft mask and pixels in the first skin region soft mask; and
segmenting the second character into the second hair region soft mask and the second skin region soft mask based on a second pixelwise difference between the second head soft mask and the second skin region soft mask.

9. A method for generating a stylized image, the method comprising:
obtaining, by a processor, data indicating a target image and a style exemplar image, the target image including a first character and the style exemplar image including a second character and wherein the style exemplar image includes a style or texture;
generating, by the processor, an appearance guide for the style exemplar image and the target image by:
generating a co

Assignees

• Adobe Inc

Inventors

• Simons David
• Lukac Michal
• Sykora Daniel
• Shechtman Elya
• Asente Paul
• Lu Jingwan
• Fiser Jakub
• Jamriska Ondrej

Classifications

• G06T11/10

  Texturing; Colouring; Generation of textures or colours (retouching, inpainting or scratch removal G06T5/77) · CPC title

• G06T15/04

  Texture mapping · CPC title

• G06T2207/20076

  Probabilistic image processing · CPC title

• G06T7/90

  Determination of colour characteristics · CPC title

• G06T2207/10016

  Video; Image sequence · CPC title