Compute amortization heuristics for lighting estimation for augmented reality

Patent metadata

Field	Value
Publication number	US-11288844-B2
Application number	US-201915733376-A
Country	US
Kind code	B2
Filing date	Oct 16, 2019
Priority date	Oct 16, 2019
Publication date	Mar 29, 2022
Grant date	Mar 29, 2022

Abstract

Systems, methods, and computer program products are described that implement obtaining, at an electronic computing device and for at least one image of a scene rendered in an Augmented Reality (AR) environment, a scene lighting estimation captured at a first time period. The scene lighting estimation may include at least a first image measurement associated with the scene. The implementations may include determining, at the electronic computing device, a second image measurement associated with the scene at a second time period, determining a function of the first image measurement and the second image measurement. Based on the determined function, the implementations may also include triggering calculation of a partial lighting estimation update or triggering calculation of a full lighting estimation update and rendering, on a screen of the electronic computing device and for the scene, the scene using the partial lighting estimation update or the full lighting estimation update.

First claim

What is claimed is:

1. A computer-implemented method, the method comprising:
obtaining, at an electronic computing device and for at least one image of a scene rendered in an Augmented Reality (AR) environment, a scene lighting estimation captured at a first time period, the scene lighting estimation including at least a first image measurement associated with the scene;
determining, at the electronic computing device, a second image measurement associated with the scene at a second time period, the second time period occurring after the first time period;
triggering, at the electronic computing device, calculation of a partial lighting estimation update in response to a difference between a first pose associated with the first image measurement and a second pose associated with the second image measurement being below a predefined threshold and in response to an elapsed time between the first time period and the second time period being below a threshold elapsed time, the first image measurement and the second image measurement include respective 6-DoF pose measurements associated with the electronic computing device depicting the scene;
triggering calculation of a full lighting estimation update in response to a function determined using the first image measurement and the second image measurement; and
rendering, on a screen of the electronic computing device and for the scene, the scene using the partial lighting estimation update or the full lighting estimation update, the partial lighting estimation update including a directional light configuration modification for display in the scene.

2. The method of claim 1 , wherein:
the first image measurement and the second image measurement include respective ambient light intensity measurements for the scene, and
the calculation of the full lighting estimation update is triggered if the determined function is above the predefined threshold and the elapsed time between the first time period and the second time period is above the threshold elapsed time, the full lighting estimation update including calculation of a High Dynamic Range (HDR) lighting estimate for the scene for an upcoming time period.

3. The method of claim 2 , wherein:
the function is a ratio of the first image measurement to the second image measurement, and
the full lighting estimation update uses at least one neural network to calculate the HDR lighting estimate for the scene.

4. The method of claim 1 , wherein the partial lighting estimation update represents linearly interpolated lighting estimations that generate gradual illumination changes to the scene according to the function.

5. The method of claim 1 , wherein the partial lighting estimation update estimates an average light intensity for the at least one image.

6. The method of claim 1 , further comprising determining an occurrence of a pose change, associated with the scene, from the first time period to the second time period;
in response to determining if the pose change is above a rotation threshold level or above a translation threshold level, triggering calculation of a High Dynamic Range (HDR) lighting estimate; and
generating an updated lighting estimation for the scene;
rendering the scene according to the updated lighting estimation.

7. The method of claim 6 , wherein the rotation threshold level and the translation threshold level configure, for the electronic computing device, a calculation frequency in which to generate updated High Dynamic Range (HDR) lighting estimates for illuminating the scene with at least one light source.

8. The method of claim 1 , wherein the full lighting estimation update represents a configuration for at least one directional light source, at least one environment map, and a plurality of spherical harmonic coefficients.

9. A computer program product tangibly embodied on a non-transitory computer-readable medium and comprising instructions that, when executed, are configured to cause at least one processor to:
obtain, for at least one image of a scene including virtual content, a scene lighting estimation captured at a first time period, the scene lighting estimation including at least a first image measurement associated with the scene;
determine a second image measurement associated with the scene at a second time period, the second time period occurring after the first time period, the first image measurement and the second image measurement include respective 6-DoF pose measurements associated with an electronic computing device depicting the scene;
determine a ratio of the first image measurement to the second image measurement;
trigger calculation of a partial lighting estimation update in response to the determined ratio being below a predefined threshold and an elapsed time between the first time period and the second time period being below a threshold elapsed time, the partial lighting estimation update including a directional light configuration modification for display in the scene;
trigger calculation of a full lighting estimation update in response to the determined ratio; and
render, for the scene, the virtual content using the partial lighting estimation update or the full lighting estimation update.

10. The computer program product of claim 9 , wherein:
the first image measurement and the second image measurement include respective ambient light intensity measurements for the scene, and
the calculation of the full lighting estimation update is triggered if the determined ratio is above the predefined threshold and the elapsed time between the first time period and the second time period is above the threshold elapsed time, the full lighting estimation update including calculation of a High Dynamic Range (HDR) lighting estimate for the scene for an upcoming time period.

11. The computer program product of claim 9 , wherein the partial lighting estimation update estimates an average light intensity for the at least one image.

12. The computer program product of claim 9 , further comprising determining an occurrence of a pose change, associated with the scene, from the first time period to the second time period;
in response to determining the pose change is above a rotation threshold level or above a translation threshold level, triggering calculation of a High Dynamic Range (HDR) lighting estimate; and
generating an updated lighting protocol for the scene;
rendering the scene according to the updated lighting protocol.

13. The computer program product of claim 12 , wherein the rotation threshold level and the translation threshold level configure, for the electronic computing device, a calculation frequency in which to generate updated High Dynamic Range (HDR) lighting estimates for illuminating the scene with at least one light source.

14. The computer program product of claim 9 , wherein the full lighting estimation update represents an illumination configuration to simulate two or more light sources.

Assignees

• Google Llc

Inventors

• Legendre Chloe
• Charbonnel Laurent
• Tong Christina
• Tsotsos Konstantine Nicholas John
• Ma Wan-Chun
• Debevec Paul

Classifications

• G06T2219/2012

  Colour editing, changing, or manipulating; Use of colour codes · CPC title

• G06T19/20

  Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts · CPC title

• G06T11/00 Primary

  Two-dimensional [2D] image generation · CPC title

• G06T19/006 Primary

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G06T3/60

  Rotation of whole images or parts thereof · CPC title