Automatically selecting example stylized images for image stylization operations based on semantic content

Patent metadata

Field	Value
Publication number	US-9594977-B2
Application number	US-201514735822-A
Country	US
Kind code	B2
Filing date	Jun 10, 2015
Priority date	Jun 10, 2015
Publication date	Mar 14, 2017
Grant date	Mar 14, 2017

Abstract

Systems and methods are provided for content-based selection of style examples used in image stylization operations. For example, training images can be used to identify example stylized images that will generate high-quality stylized images when stylizing input images having certain types of semantic content. In one example, a processing device determines which example stylized images are more suitable for use with certain types of semantic content represented by training images. In response to receiving or otherwise accessing an input image, the processing device analyzes the semantic content of the input image, matches the input image to at least one training image with similar semantic content, and selects at least one example stylized image that has been previously matched to one or more training images having that type of semantic content. The processing device modifies color or contrast information for the input image using the selected example stylized image.

First claim

The invention claimed is:

1. A method for automatically selecting and applying an image stylization operation based on semantic content of a received image, the method comprising:
determining, by a processor, that color information or contrast information of a training image is similar to color information or contrast information of an example stylized image;
matching, by a processor, the training image to an input image that is semantically similar to the training image;
selecting, by a processor, the example stylized image based on the example stylized image and the training image having similar color information or contrast information; and
modifying, by a processor, color information or contrast information of the input image based on the color information or contrast information from the selected example stylized image.

2. The method of claim 1 , wherein the example stylized image is selected based on determining that the color information or contrast information of the example stylized image is similar to color information or contrast information of a sufficiently high number of images from a cluster of semantically similar training images that includes the training image.

3. The method of claim 1 , wherein determining that the color information or contrast information of training image is similar to the color information or contrast information of an example stylized image comprises:
grouping a subset of training images from a set of training images into a cluster based on a semantic similarity of shapes and spatial relationships depicted in the grouped training images, wherein the subset of training images includes the training image;
comparing the color information or contrast information of the example stylized image to color information or contrast information of each training image in the subset of training images; and
determining that a sufficiently large number of training images in the subset of training images has color information or contrast information that is similar to the color information or contrast information of the example stylized image.

4. The method of claim 1 , wherein the color information comprises a distribution of colors and the contrast information comprises at least one of global contrast, local contrast, lighting direction, and vignetting.

5. The method of claim 1 , wherein determining that the training image and the example stylized image have similar color information or contrast information comprises determining a style ranking for the example stylized image with respect to a cluster of semantically similar training images that include the training image, wherein the style ranking is determined based on (i) respective Gaussian statistics for respective chrominance channels of the example stylized image and the training images in the cluster and (ii) respective Euclidean distances between a vector representing a luminance channel of the example stylized image and respective additional vectors representing additional luminance channels of the training images in the cluster.

6. The method of claim 1 , wherein determining that the training image and the example stylized image have similar color information or contrast information comprises determining a style ranking for the example stylized image with respect to a cluster of semantically similar training images that include the training image, wherein determining the style ranking comprises:
determining stylistic similarity metrics for the example stylized image with respect to the training images in the cluster, wherein determining each stylistic similarity metric for a respective training image comprises:
determining first Gaussian statistics for a first chrominance channel of the example stylized image and second Gaussian statistics for a second chrominance channel of the respective training image,
calculating a respective Hellinger distance based on the first Gaussian statistics and the second Gaussian statistics,
determining a first luminance vector representing a first luminance channel of the example stylized image and a second luminance vector representing a second luminance channel of the respective training image, and
calculating a respective Euclidean distance based on the first luminance vector and the second luminance vector,
wherein the stylistic similarity metric is determined from the respective Hellinger distance and the respective Euclidean distance; and

determining the style ranking by aggregating the stylistic similarity metrics.

7. The method of claim 1 , wherein the method further comprises:
selecting sufficiently diverse example stylized images for stylizing the input image, wherein selecting the sufficiently diverse example stylized images comprises:
selecting a plurality of example stylized images that have sufficiently high style rankings with respect to a cluster of semantically similar training images that include the training image,
determining that a first example stylized image and a second example stylized image from the plurality of example stylized images have an excessive stylistic similarity with respect to one another, and
selecting a subset of example stylized images from the plurality of example stylized images that excludes at least one of the first example stylized image and the second example stylized image; and

modifying color information or contrast information of the input image based on color information or contrast information of the selected subset of example stylized images.

8. The method of claim 7 , wherein the excessive stylistic similarity is determined based on first Gaussian statistics for a first chrominance channel of the first example stylized image and second Gaussian statistics for a second chrominance channel of the first example stylized image.

9. The method of claim 1 , further comprising:
grouping subsets of training images from a set of training images into clusters, wherein each subset of training images is grouped into a respective cluster on a semantic similarity of shapes and spatial relationships depicted in the grouped training images, wherein a similarity of color information or contrast information of the training image and the example stylized image is determined based on a sufficiently large number of training images in a first cluster that includes the training image having color information or contrast information that is similar to color information or contrast information of the example image;
determining that a sufficiently large number of training images in a second cluster that includes an additional training image have color information or contrast information that is similar to color information or contrast information of an additional example stylized image;
matching the additional training image from the second cluster to the input image based on the additional training image being semantically similar to the input image;
selecting the additional example stylized image based on (i) the additional example stylized image and the additional training image having similar color information or contrast information and (ii) the example stylized image and the additional example stylized image having sufficiently different color information or contrast information
modifying color information or contrast information of the input image based on color information or contrast information from the additional example stylized image.

10. The method of claim 9 , wherein the subsets of training images are grouped into the clusters prior to receiving the input image.

11. The method of claim 1 , further comprising at least one of:
selecting the example stylized image based on a prior user input indicating a preference for

Assignees

• Adobe Systems Inc

Inventors

• Lin Zhe
• Shen Xiaohui
• Sunkavalli Kalyan
• Lee Joon-Young

Classifications

• G06T7/90 Primary

  Determination of colour characteristics · CPC title

• G06T7/60

  Analysis of geometric attributes · CPC title

• G06T11/10

  Texturing; Colouring; Generation of textures or colours (retouching, inpainting or scratch removal G06T5/77) · CPC title

• G06V20/00

  Scenes; Scene-specific elements (control of digital cameras H04N23/60) · CPC title

• G06K9/6256

  Physics · mapped topic