Image color and tone style transfer

Patent metadata

Field	Value
Publication number	US-9857953-B2
Application number	US-201514944019-A
Country	US
Kind code	B2
Filing date	Nov 17, 2015
Priority date	Nov 17, 2015
Publication date	Jan 2, 2018
Grant date	Jan 2, 2018

Abstract

In embodiments of image color and tone style transfer, a computing device implements an image style transfer algorithm to generate a modified image from an input image based on a color style and a tone style of a style image. A user can select the input image that includes color features, as well as select the style image that includes an example of the color style and the tone style to transfer to the input image. A chrominance transfer function can then be applied to transfer the color style to the input image, utilizing a covariance of an input image color of the input image to control modification of the input image color. A luminance transfer function can also be applied to transfer the tone style to the input image, utilizing a tone mapping curve based on a non-linear optimization to estimate luminance parameters of the tone mapping curve.

First claim

The invention claimed is:

1. A method to transfer a color and tone style of a style image to an input image, the method comprising:
receiving a selection of the input image that includes color features;
receiving a selection of the style image that includes an example of the color and tone style to transfer to the input image;
generating a modified image from the input image based on the color and tone style of the style image, the generating performed by an image style transfer algorithm executed by a memory and processor system of a computing device, the generating comprising:
applying a chrominance transfer function to transfer the color style of the style image to the input image, the chrominance transfer function implementing a covariance of an input image color of the input image to control modification of the input image color; and
applying a luminance transfer function to transfer the tone style of the style image to the input image, the luminance transfer function implementing a tone mapping curve based on a non-linear optimization to estimate luminance parameters of the tone mapping curve.

2. The method as recited in claim 1 , wherein said generating the modified image based on the color and tone style of the style image without creating visual artifacts of the color features in the modified image by taking into account the input image color and an input image tone.

3. The method as recited in claim 1 , further comprising, prior to said applying the chrominance transfer function and said applying the luminance transfer function:
compressing dynamic ranges of the style image and the input image into a colorspace that decorrelates a luminance component of the tone style from a chrominance component of the color style of the style image;
expanding the luminance component over a full dynamic range;
clipping a designated minimum and a maximum percentage of luminance levels over the full dynamic range; and
said applying the chrominance transfer function to the chrominance component and said applying the luminance transfer function to the luminance component to generate the modified image.

4. The method as recited in claim 1 , wherein image content in the style image is different than in the input image, the different image content providing the color and tone style to transfer to the input image.

5. The method as recited in claim 1 , wherein the chrominance transfer function is controlled with a chrominance scaling maximum to prevent an over-amplification of the color features in the input image to generate the modified image.

6. The method as recited in claim 1 , wherein the chrominance transfer function is implemented to:
compute a chrominance of the input image;
compute a chrominance of the style image; and
manipulate the computed chrominance of the input image to approximate the computed chrominance of the style image.

7. The method as recited in claim 1 , further comprising:
receiving a user-selectable input to initiate an adjustment of an output chrominance level of the chrominance transfer function, wherein a regularization term of the chrominance transfer function is manipulated to effectuate the adjustment of the output chrominance level applied to the input image to generate the modified image.

8. The method as recited in claim 1 , wherein the luminance transfer function computes the tone mapping curve to approximately match a histogram of the input image luminance to a histogram of the style image luminance.

9. The method as recited in claim 1 , further comprising:
receiving a user-selectable input to initiate an adjustment of an output luminance level of the luminance transfer function, wherein terms of the luminance transfer function are manipulated to effectuate the adjustment of the output luminance level applied to the input image to generate the modified image.

10. A computing device implemented to transfer a color and tone style of a style image to an input image, the computing device comprising:
a display device configured to display the input image that includes color features;
a memory and processor system configured to execute an image style transfer algorithm as a computer application implemented to generate a modified image from the input image based on the color and tone style of the style image, the image style transfer algorithm configured to:
apply a chrominance transfer function to transfer the color style of the style image to the input image, the chrominance transfer function implementing a covariance of an input image color of the input image to control modification of the input image color; and
apply a luminance transfer function to transfer the tone style of the style image to the input image, the luminance transfer function implementing a tone mapping curve based on a non-linear optimization to estimate luminance parameters of the tone mapping curve.

11. The computing device as recited in claim 10 , wherein the image style transfer algorithm is further implemented to said generate the modified image based on the color and tone style of the style image without creating visual artifacts of the color features in the modified image by taking into account the input image color and an input image tone.

12. The computing device as recited in claim 10 , wherein the image style transfer algorithm is further implemented to:
compress dynamic ranges of the style image and the input image into a colorspace that decorrelates a luminance component of the tone style from a chrominance component of the color style of the style image;
expand the luminance component over a full dynamic range;
clip a designated minimum and a maximum percentage of luminance levels over the full dynamic range; and
said apply the chrominance transfer function to the chrominance component and said apply the luminance transfer function to the luminance component to generate the modified image.

13. The computing device as recited in claim 10 , wherein the chrominance transfer function is controlled with a chrominance scaling maximum to prevent an over-amplification of the color features in the input image to generate the modified image.

14. The computing device as recited in claim 10 , wherein the chrominance transfer function is implemented to:
compute a chrominance of the input image;
compute a chrominance of the style image; and
manipulate the computed chrominance of the input image to approximate the computed chrominance of the style image.

15. The computing device as recited in claim 10 , wherein luminance transfer function computes the tone mapping curve to approximately match a histogram of the input image luminance to a histogram of the style image luminance.

16. The computing device as recited in claim 10 , further comprising an image style user interface configured to display the modified image, the image style user interface comprising:
a chrominance user-selectable control to initiate an adjustment of an output chrominance level of the chrominance transfer function, wherein a regularization term of the chrominance transfer function is manipulated to effectuate the adjustment of the output chrominance level applied to the input image to generate the modified image; and
a luminance user-selectable control to initiate an adjustment of an output luminance level of the luminance transfer function, wherein terms of the luminance transfer function are manipulated to effectuate the adjustment of the output luminance level applied to the input image to generate the modified image.

17. A method for image color and tone style transfer, the method c

Assignees

• Adobe Systems Inc

Inventors

• Sunkavalli Kalyan K
• Lin Zhe
• Shen Xiaohui
• Lee Joon-Young

Classifications

• G06T11/10

  Texturing; Colouring; Generation of textures or colours (retouching, inpainting or scratch removal G06T5/77) · CPC title

• G06F3/04845 Primary

  for image manipulation, e.g. dragging, rotation, expansion or change of colour · CPC title

• G06T11/001

  Physics · mapped topic

• G06F3/04842

  Selection of displayed objects or displayed text elements (G06F3/0482 takes precedence) · CPC title

• G06F3/0488

  using a touch-screen or digitiser, e.g. input of commands through traced gestures · CPC title