Image lighting transfer via multi-dimensional histogram matching

Patent metadata

Field	Value
Publication number	US-10521892-B2
Application number	US-201615253655-A
Country	US
Kind code	B2
Filing date	Aug 31, 2016
Priority date	Aug 31, 2016
Publication date	Dec 31, 2019
Grant date	Dec 31, 2019

Abstract

Embodiments of the present invention provide systems, methods, and computer storage media directed at relighting a target image based on a lighting effect from a reference image. In one embodiment, a target image and a reference image are received, the reference image includes a lighting effect desired to be applied to the target image. A lighting transfer is performed using color data and geometrical data associated with the reference image and color data and geometrical data associated with the target image. The lighting transfer causes generation of a relit image that corresponds with the target image having a lighting effect of the reference image. The relit image is provided for display to a user via one or more output devices. Other embodiments may be described and/or claimed.

First claim

What is claimed is:

1. One or more non-transitory computer-readable storage media having instructions stored thereon, which, when executed by one or more processors of a computing device, cause the computing device to perform actions comprising:
receiving a target image and a reference image, the reference image including a lighting effect desired to be applied to the target image;
employing a multi-dimensional histogram matching technique that applies a transfer function to the target image, wherein the transfer function comprises a probability density function and is based on a combination of color data and three-dimensional (3D) geometrical data associated with the reference image and a combination of color data and 3D geometrical data associated with the target image, and wherein applying the transfer function to the target image effectuates a lighting transfer that causes a geometry-based transfer of a lighting effect from the reference image to a relit image that corresponds with the target image, such that the relit image includes the lighting effect that was transferred from the reference image;
providing the relit image for display to a user via one or more output devices.

2. The one or more non-transitory computer-readable storage media of claim 1 , wherein the actions further comprise:
generating the color data and geometrical data associated with the reference image.

3. The one or more non-transitory computer-readable storage media of claim 1 , wherein the actions further comprise:
generating the color data and geometrical data associated with the target image.

4. The one or more non-transitory computer-readable storage media of claim 1 , wherein the color data associated with the reference image and the color data associated with the target image comprises Lab color values.

5. The one or more non-transitory computer-readable storage media of claim 1 , wherein the geometrical data associated with the reference image and the geometrical data associated with the target image comprises coordinate data and surface orientation data.

6. The one or more non-transitory computer-readable storage media of claim 1 , wherein the reference image comprises a photograph, a three-dimensional rendered image, or a user-edited image.

7. The one or more non-transitory computer-readable storage media of claim 1 , wherein the actions further comprise:
generating a multi-dimensional feature vector for the reference image that includes the color data and the geometrical data associated with the reference image;
generating a multi-dimensional feature vector for the target image that includes the color data and the geometrical data associated with the target image; and
matching the multi-dimensional feature vector for the reference image with the multi-dimensional feature vector for the target image to transfer the lighting.

8. One or more non-transitory computer-readable storage media having instructions stored thereon, which, when executed by one or more processors of a computing device, cause the computing device to perform actions comprising:
receiving a target image and a reference image, the reference image including a lighting effect desired to be applied to the target image;
generating reference image data for pixels of the reference image, the reference image data including reference three-dimensional (3D) geometrical data and reference color data for the respective pixels of the reference image;
generating target image data for pixels of the target image, the target image data including target 3D geometrical data and target color data for the respective pixels of the target image;
employing a multi-dimensional histogram matching technique that applies a transfer function to the target image, wherein the transfer function comprises a probability density function and is based on a combination of the reference color data and the reference 3D geometrical data associated with the reference image and a combination of the target color data and the target 3D geometrical data associated with the target image and wherein applying the transfer function to the target image effectuates a lighting transfer that causes a geometry-based transfer of a lighting effect from the reference image to a relit image that corresponds with the target image, such that the relit image includes the lighting effect that was transferred from the reference image; and
providing the relit image for display the target image that includes the lighting effect that was transformed from the reference image.

9. The one or more non-transitory computer-readable storage media of claim 8 , wherein the reference image and the target image are input images designated by a user.

10. The one or more non-transitory computer-readable storage media of claim 8 , wherein the reference color data and the target color data comprise Lab color values.

11. The one or more non-transitory computer-readable storage media of claim 8 , wherein the reference 3D geometrical data and the target 3D geometrical data comprise coordinates and surface orientation data.

12. The one or more non-transitory computer-readable storage media of claim 11 , wherein the coordinates comprise local coordinates.

13. The one or more non-transitory computer-readable storage media of claim 11 , wherein the surface orientation data comprises surface normal data.

14. The one or more non-transitory computer-readable storage media of claim 8 , wherein the actions further comprise:
generate multi-dimensional vectors associated with pixels in the reference image using the reference 3D geometrical data and the reference color data; and
generate multi-dimensional vectors associated with pixels in the target image using the target 3D geometrical data and the target color data.

15. The one or more non-transitory computer-readable storage media of claim 14 , wherein the multi-dimensional vectors are used to perform the lighting transfer via multi-dimensional histogram matching.

16. A computer-implemented method for facilitating image relighting, the method comprising:
receiving a target image and a reference image, the reference image including a lighting effect desired to be applied to the target image;
generating reference image data for pixels of the reference image, the reference image data including reference three-dimensional (3D) geometrical data and reference color data for the respective pixels of the reference image;
generating target image data for pixels of the target image, the target image data including target 3D geometrical data and target color data for the respective pixels of the target image;
employing a multi-dimensional histogram matching technique that applies a transfer function to the target image, wherein the transfer function comprises a probability density function and is based on a combination of the reference color data and the reference 3D geometrical data associated with the reference image and a combination of the target color data and the target 3D geometrical data associated with the target image, and wherein applying of the transfer function to the target image effectuates a lighting transfer that causes a geometry-based transfer of a lighting effect from the reference image to a relit image that corresponds with the target image, such that the relit image includes the lighting effect that was transferred from the reference image; and
outputting the target image to the user via one or more output devices.

17. The method of claim 16 further comprising fitting a three-dimensional model to an object in the target image and a thr

Assignees

• Adobe Inc

Inventors

• Sunkavalli Kalyan K
• Hadap Sunil
• Shechtman Elya
• Shu Zhixin

Classifications

• G06T5/50 Primary

  using two or more images, e.g. averaging or subtraction · CPC title

• G06T2207/10024

  Color image · CPC title

• G06T11/00

  Two-dimensional [2D] image generation · CPC title