Dynamic lighting for objects in images

Patent metadata

Field	Value
Publication number	US-10643375-B2
Application number	US-201816107868-A
Country	US
Kind code	B2
Filing date	Aug 21, 2018
Priority date	Feb 26, 2018
Publication date	May 5, 2020
Grant date	May 5, 2020

Abstract

Techniques and systems are described herein for determining dynamic lighting for objects in images. Using such techniques and systems, a lighting condition of one or more captured images can be adjusted. Techniques and systems are also described herein for determining depth values for one or more objects in an image. In some cases, the depth values (and the lighting values) can be determined using only a single camera and a single image, in which case one or more depth sensors are not needed to produce the depth values.

First claim

What is claimed is:

1. A method for adjusting a lighting condition in one or more captured images, the method comprising:
detecting feature points of an object in an image;
generating an object-specific model including mapped points, the mapped points being determined by fitting points of a three-dimensional model to the feature points of the object detected in the image;
determining depth values for the mapped points of the object-specific model;
generating an object-specific depth map associated with the object, the object-specific depth map including the depth values determined for the mapped points of the object-specific model;
determining, using the object-specific depth map and based on user input associated with one or more lighting characteristics for the image, lighting values for points of the object in the image; and
applying the lighting values to the points of the object in the image.

2. The method of claim 1 , further comprising:
obtaining a segmentation mask including foreground pixels representing the object in the image; and
determining, using the segmentation mask, a region of the image including the object.

3. The method of claim 1 , further comprising:
obtaining coordinates of the feature points of the object in the image;
determining coordinates of corresponding points of the three-dimensional model that correspond to the feature points of the object;
determining the mapped points of the object-specific model by modifying the coordinates of the corresponding points of the three-dimensional model to match the coordinates of the feature points of the object; and
modifying coordinates of remaining points of the three-dimensional model to match coordinates of remaining points of the object in the image by interpolating between the feature points.

4. The method of claim 1 , further comprising selecting the three-dimensional model from a plurality of models based on the object in the image.

5. The method of claim 4 , wherein the plurality of models are stored locally on a computing device.

6. The method of claim 4 , wherein the plurality of models are stored remotely on one or more servers.

7. The method of claim 1 , further comprising:
receiving the image of the object; and
generating a segmentation mask by segmenting the image into foreground pixels of the object and background pixels.

8. The method of claim 1 , further comprising:
storing the object-specific model.

9. The method of claim 8 , further comprising:
receiving an additional image of the object;
obtaining the object-specific model of the object; and
determining lighting values for points of the object in the additional image using the object-specific model.

10. The method of claim 1 , wherein generating the object-specific depth map includes using a segmentation mask to determine which pixels from the image include foreground pixels of the object.

11. The method of claim 1 , wherein the one or more lighting characteristics include one or more of an intensity of a light source, a color of light from the light source, a direction of the light source relative to the object in the image, a position of the light source relative to the object in the image, a diffuseness of the light source, and a roughness of a surface of the object.

12. The method of claim 1 , wherein determining the lighting values for the points of the object in the image includes:
determining a normal map using the object-specific depth map, the normal map including vectors perpendicular to a surface of the object;
determining, based on the user input, one or more characteristics of a light source, the one or more characteristics of the light source including one or more of an intensity of the light source, a color of light from the light source, a direction of the light source relative to the object in the image, a position of the light source relative to the object in the image, and a diffuseness of the light source;
determining a reflectivity of each point of the object in the image based on a roughness of the object; and
determining the lighting values for the points of the object in the image based on the normal map, the one or more characteristics of the light source, and the reflectivity of each point of the object.

13. The method of claim 1 , wherein the object includes a face of a person.

14. The method of claim 13 , wherein the feature points of the object in the image include facial landmarks on the face.

15. The method of claim 1 , wherein the image of the object is captured using a single camera, and wherein the depth values are determined without using an additional camera other than the single camera, without using an additional image other than the image of the object, and without using any depth sensor.

16. The method of claim 1 , further comprising:
receiving, from a graphical user interface, the user input associated with the one or more lighting characteristics for the image; and
displaying the image of the object with the applied lighting values.

17. The method of claim 16 , wherein the user input is received in response to an adjustment of a slider graphical element of the graphical user interface, in response to selection of an icon graphical element of the graphical user interface, in response to entry of text in a text entry graphical element of the graphical user interface, in response to an audio command from the user, or in response to a gesture command from the user.

18. The method of claim 1 , further comprising:
receiving additional user input requesting a change in lighting;
determining, based on the additional user input, updated lighting values for the points of the object in the image; and
applying the updated lighting values to the points of the object in the image.

19. The method of claim 1 , wherein the lighting values are determined without using pre-determined lighting values determined prior to receiving the image.

20. An apparatus for adjusting a lighting condition in one or more captured images, comprising:
a memory configured to store one or more images; and
one or more processors configured to:
detect feature points of an object in an image;
generate an object-specific model including mapped points, the mapped points being determined by fitting points of a three-dimensional model to the feature points of the object detected in the image;
determine depth values for the mapped points of the object-specific model;
generate an object-specific depth map associated with the object, the object-specific depth map including the depth values determined for the mapped points of the object-specific model;
determine, using the object-specific depth map and based on user input associated with one or more lighting characteristics for the image, lighting values for points of the object in the image; and
apply the lighting values to the points of the object in the image.

21. The apparatus of claim 20 , wherein the one or more processors are configured to:
obtain a segmentation mask including foreground pixels representing the object in the image; and
determine, using the segmentation mask, a region of the image including the object.

22. The apparatus of claim 20 , wherein the one or more processors are configured to:
obtain coordinates of the feature points of the object in the image;
determine coordinates of corresponding points of the three-dimensional model that correspond to the feature points of the object;
determining the mapped points of

Assignees

• Qualcomm Inc

Inventors

• Rezaiifar Ramin
• Gaggi Antwan
• Roy Donna

Classifications

• G06T7/596

  from three or more stereo images · CPC title

• G06F3/167

  Audio in a user interface, e.g. using voice commands for navigating, audio feedback · CPC title

• G06T7/586

  from multiple light sources, e.g. photometric stereo · CPC title

• G06T2207/10028

  Range image; Depth image; 3D point clouds · CPC title

• G06T7/194

  involving foreground-background segmentation · CPC title