Depth-based relighting in augmented reality

Patent metadata

Field	Value
Publication number	US-12469227-B2
Application number	US-202018000237-A
Country	US
Kind code	B2
Filing date	Jun 22, 2020
Priority date	Jun 22, 2020
Publication date	Nov 11, 2025
Grant date	Nov 11, 2025

Abstract

A system and method for relighting of an augmented reality scene are provided. Image data and depth data corresponding to a physical environment may be captured by an image sensor of a computing device, for display in an AR scene generated by the computing device. The system may use the image data and depth data to generate a live depth map of the physical environment. A virtual light source may be placed in the AR scene of the physical environment in response to a user input. The system may relight the AR scene in response to the placement of the virtual light source based on the image data and the depth map.

First claim

What is claimed is:

1 . A computer-implemented method, comprising:
capturing, by an image sensor of a computing device, image data corresponding to an image of a physical environment; detecting, by a depth sensor, depth data corresponding to the physical environment; generating, by a processor, an augmented reality (AR) scene from the image data corresponding to the image of the physical environment; receiving, at an input device of the computing device, an input for a position of a virtual light source in the AR scene; determining a two-dimensional position of the input received at the input device; determining a three-dimensional position, corresponding to a placement position of the virtual light source in the AR scene, the three-dimensional position corresponding to the two-dimensional position of the input received at the input device; placing a virtual light source in the AR scene at the three-dimensional position; and relighting an object in the AR scene, and at least a portion of the physical environment adjacent the object, based on the three-dimensional position of the virtual light source in the AR scene and the depth data corresponding to the physical environment.

2 . The computer-implemented method of claim 1 , wherein detecting the depth data includes generating a depth map corresponding to the physical environment.

3 . The computer-implemented method of claim 2 , wherein relighting the AR scene includes:
for each pixel, of a plurality of pixels of the AR scene:
setting an intensity level of the pixel based on an intensity of the virtual light source and a distance of the pixel from the virtual light source in the AR scene;
setting a color of the pixel based on a color of the virtual light source, the distance of the pixel from the virtual light source, and a color of the pixel in the AR scene; and
setting a reflectivity of the pixel based on the color of the virtual light source and the color of the pixel in the AR scene.

4 . The computer-implemented method of claim 3 , wherein relighting the AR scene includes, for each pixel of the plurality of pixels of the AR scene:
detecting obstacles between the placement position of the virtual light source and the pixel based on the depth map; and adjusting the intensity level, the color, and the reflectivity of the pixel based on the obstacles detected between the placement position of the virtual light source and the pixel.

5 . The computer-implemented method of claim 4 , wherein relighting the AR scene includes:
applying the adjusted intensity level, the adjusted color, and the adjusted reflectivity, to the AR scene; generating a composite rendering of the physical environment in the AR scene including the adjusted intensity level, the adjusted color, and the adjusted reflectivity; and displaying the composite rendering of the AR scene on a display device of the computing device.

6 . The computer-implemented method of claim 3 , wherein relighting the AR scene includes, for each pixel of the plurality of pixels:
detecting a distance of the pixel from the placement position of the virtual light source in the AR scene; and in response to detection of the pixel at a position corresponding to the placement position of the virtual light source in the AR scene, setting the intensity, the color and the reflectivity of the pixel to the intensity, the color and the reflectivity of the virtual light source.

7 . The computer-implemented method of claim 6 , wherein the intensity of the pixel is inversely proportional to a square of the distance between the virtual light source and the pixel.

8 . The computer-implemented method of claim 1 , further comprising:
detecting, by the image sensor, color data corresponding to the image of the physical environment; and generating a color map corresponding to the image of the physical environment, the color map providing at least one of color characteristics or reflectivity characteristics of each pixel of a plurality of pixels of the AR scene of the physical environment.

9 . The computer-implemented method of claim 1 , wherein generating the AR scene includes displaying the image of the physical environment in a darkened state, including darkening the image of the object and at least a portion of the physical environment adjacent the object to simulate a darkened physical environment without ambient lighting.

10 . The computer-implemented method of claim 9 , wherein relighting the AR scene includes relighting the AR scene generated from the darkened image of the physical environment, including, for each pixel of a plurality of pixels located between the virtual light source and the object, relighting the pixel to at least one of an adjusted intensity level, an adjusted color, and an adjusted reflectivity.

11 . The computer-implemented method of claim 9 , wherein relighting the AR scene includes relighting the AR scene generated from the darkened image of the physical environment to simulate illumination of the physical environment by the virtual light source without the influence of the ambient lighting.

12 . The computer-implemented method of claim 1 , further comprising receiving, at the input device of the computing device, an input placing a second virtual light source in the AR scene;
wherein the relighting the AR scene includes relighting the AR scene based on the placement position of the virtual light source in the AR scene and the placement position of the second virtual light source in the AR scene.

13 . The computer-implemented method of claim 1 , wherein relighting the AR scene includes relighting a portion of the AR scene located between the virtual light source and the object.

14 . A system, comprising:
a camera assembly; a display device; at least one processor; and a memory storing instructions that, when executed by the at least one processor, cause the system to:
capture image data corresponding to an image of a physical environment;
detect depth data corresponding to the physical environment;
generate an augmented reality (AR) scene from the image data corresponding to the image of the physical environment;
receive an input at a position on an input portion of the display device displaying the AR scene corresponding to a selected placement position of a virtual light source in the AR scene; and
relight an object in the AR scene, and at least a portion of the physical environment adjacent the object, based on the placement position of the virtual light source in the AR scene and the depth data corresponding to the physical environment.

15 . The system of claim 14 , wherein the camera assembly includes a depth sensor and an image sensor, and wherein the instructions cause the system to at least one of:
generate a depth map of the physical environment based on the depth data collected by the depth sensor; or generate a color map of the physical environment based on the image data collected by the image sensor, the color map providing at least one of color characteristics or reflectivity characteristics of each pixel of a plurality of pixels of the AR scene of the physical environment.

16 . The system of claim 15 , wherein the instructions cause the system to:
detect a three-dimensional position of the virtual light source in the physical environment corresponding to the placement position of the virtual light source in the AR scene; and for each pixel, of a plurality of pixels of the AR scene:
set an intensity level of the pixel based on an intensity of the virtual light source and a distance of the pixel from the

Assignees

• Google Llc

Inventors

• Du Ruofei
• Kim David

Classifications

• G06T2219/2012

  Colour editing, changing, or manipulating; Use of colour codes · CPC title

• G06T2207/10028

  Range image; Depth image; 3D point clouds · CPC title

• G06T19/20

  Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts · CPC title

• G06T15/506 Primary

  Illumination models · CPC title

• G06T7/70

  Determining position or orientation of objects or cameras (camera calibration G06T7/80) · CPC title