Enhanced photo relighting based on machine learning models

Patent metadata

Field	Value
Publication number	US-12412255-B2
Application number	US-202118028930-A
Country	US
Kind code	B2
Filing date	May 17, 2021
Priority date	Sep 30, 2020
Publication date	Sep 9, 2025
Grant date	Sep 9, 2025

Abstract

Apparatus and methods related to applying lighting models to images of objects are provided. An example method includes applying a geometry model to an input image to determine a surface orientation map indicative of a distribution of lighting on an object based on a surface geometry. The method further includes applying an environmental light estimation model to the input image to determine a direction of synthetic lighting to be applied to the input image. The method also includes applying, based on the surface orientation map and the direction of synthetic lighting, a light energy model to determine a quotient image indicative of an amount of light energy to be applied to each pixel of the input image. The method additionally includes enhancing, based on the quotient image, a portion of the input image. One or more neural networks can be trained to perform one or more of the aforementioned aspects.

First claim

What is claimed is:

1. A computer-implemented method, comprising:
determining, by a geometry model of a computing device, a surface orientation map based on a prediction of lighting on an object in an input image based on a surface geometry of the object;
determining, by an environmental light estimation model of the computing device, a direction of synthetic lighting to be applied to the input image to modify at least a portion of the input image;
determining, by a light energy model of the computing device and based on the surface orientation map and the direction of synthetic lighting, a quotient image indicative of an amount of light energy to be applied to each pixel of the input image; and
modifying, based on a product of the quotient image and the input image, the portion of the input image.

2. The computer-implemented method of claim 1 , wherein the determining of the direction of synthetic lighting comprises:
detecting, by the computing device, a pose of the object in the input image, and
wherein the determining of the direction of the synthetic lighting is based on the pose.

3. The computer-implemented method of claim 1 , further comprising:
generating, by the computing device and based on the surface orientation map and the direction of synthetic lighting, a light visibility map, and
wherein the determining of the quotient image is based on the light visibility map.

4. The computer-implemented method of claim 1 , wherein the modifying of the portion of the input image comprises:
determining, by the computing device, a request to modify the portion of the input image;
sending the request to request to modify the portion of the input image from the computing device to a second computing device, the second computing device comprising a trained neural network; and
after sending the request, the computing device receiving, from the second computing device, an output image that applies the quotient image to modify the portion of the input image.

5. The computer-implemented method of claim 1 , wherein the determining of the direction of synthetic lighting comprises:
receiving, by the computing device, a user preference for the direction of the synthetic lighting to be applied to the input image.

6. The computer-implemented method of claim 1 , wherein the object has a characteristic of diffusely reflecting light.

7. The computer-implemented method of claim 1 , wherein the computing device comprises a camera, and the method further comprising:
generating the input image of the object using the camera; and
receiving, at the computing device, the generated input image from the camera.

8. The computer-implemented method of claim 1 , further comprising:
providing the portion of the modified input image using the computing device.

9. The computer-implemented method of claim 1 , further comprising:
predicting an illumination profile of the input image using the environmental light estimation model; and
providing the predicted illumination profile using the computing device.

10. The computer-implemented method of claim 9 , wherein the prediction of the illumination profile comprises:
generating a high dynamic range (HDR) lighting environment based on low dynamic range (LDR) images of a plurality of reference objects, each of the reference objects having a respective bidirectional reflectance distribution function (BRDF).

11. The computer-implemented method of claim 10 , wherein the set of reference objects includes one or more of a mirror ball, a matte silver ball, or a gray diffuse ball.

12. The computer-implemented method of claim 1 , further comprising:
adjusting the quotient image to apply one or more of: (i) a first compensation for an exposure level in the input image, (ii) a second compensation for a brightness level in the input image, or (iii) a matting refinement to the input image.

13. A computing device, comprising:
one or more processors; and
data storage, wherein the data storage has stored thereon computer-executable instructions that, when executed by the one or more processors, cause the computing device to carry out functions comprising:
determining, by a geometry model of the computing device, a surface orientation map based on a prediction of lighting on an object in an input image based on a surface geometry of the object;
determining, by an environmental light estimation model of the computing device, a direction of synthetic lighting to be applied to the input image to modify at least a portion of the input image;
determining, by a light energy model of the computing device and based on the surface orientation map and the direction of synthetic lighting, a quotient image indicative of an amount of light energy to be applied to each pixel of the input image; and
modifying, based on a product of the quotient image and the input image, the portion of the input image.

14. An article of manufacture comprising one or more computer readable media having computer-readable instructions stored thereon that, when executed by one or more processors of a computing device, cause the computing device to carry out functions comprising:
determining, by a geometry model of the computing device, a surface orientation map based on a prediction of lighting on an object in an input image based on a surface geometry of the object;
determining, by an environmental light estimation model of the computing device, a direction of synthetic lighting to be applied to the input image to modify at least a portion of the input image;
determining, by a light energy model of the computing device and based on the surface orientation map and the direction of synthetic lighting, a quotient image indicative of an amount of light energy to be applied to each pixel of the input image; and
modifying, based on a product of the quotient image and the input image, the portion of the input image.

15. The computer-implemented method of claim 1 , wherein at least a part of one or more of the geometry model, the environmental light estimation model, or the light energy model is part of a machine learning model.

16. The computer-implemented method of claim 15 , further comprising:
training the machine learning model to perform one or more of: (1) predict the surface orientation map, (2) predict the direction of the synthetic lighting, or (3) predict the quotient image.

17. The computer-implemented method of claim 15 , wherein the machine learning model is trained to predict the surface orientation map, and the method further comprising:
predicting the surface orientation map by using the trained machine learning model.

18. The computer-implemented method of claim 15 , wherein the machine learning model is trained to predict the direction of the synthetic lighting, and the method further comprising:
predicting the direction of the synthetic lighting by using the trained machine learning model.

19. The computer-implemented method of claim 15 , wherein the machine learning model is trained to predict the quotient image, and the method further comprising:
predicting the quotient image by using the trained machine learning model.

20. The computer-implemented method of claim 15 , wherein the training of the machine learning model is based on a training dataset comprising a plurality of images of the object, wherein the plurality of images utilize a plurality of illumination profiles to illuminate the object.

21. The computer-implemented method of claim 15 , wherein the training of

Assignees

• Google Llc

Inventors

• Fanello Sean Ryan Francesco
• Tsai Yun-Ta
• Pandey Rohit Kumar
• Debevec Paul
• Milne Michael
• Legendre Chloe
• Barron Jonathan Tilton
• Rhemann Christoph
• Bouaziz Sofien
• Sarma Navin Padman

Classifications

• G06T11/10

  Texturing; Colouring; Generation of textures or colours (retouching, inpainting or scratch removal G06T5/77) · CPC title

• G06T2207/30201

  Face · CPC title

• G06T2207/20081

  Training; Learning · CPC title

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title

• G06T5/94

  based on local image properties, e.g. for local contrast enhancement · CPC title