Apparatus and methods for dynamic white point compensation to improve perceived color of synthetic content

Patent metadata

Field	Value
Publication number	US-11593971-B2
Application number	US-201816617982-A
Country	US
Kind code	B2
Filing date	May 16, 2018
Priority date	May 31, 2017
Publication date	Feb 28, 2023
Grant date	Feb 28, 2023

Abstract

The invention pertains to methods and apparatus for compensating the rendering of virtual content in an augmented reality scene as a function of the white point, spectral distribution, color or other optical parameter of the real content in the scene.

First claim

What is claimed is:

1. A method of displaying synthetic content on an augmented reality display device, the method comprising:
capturing an image of a scene;
estimating a white point of light illuminating the scene based on the captured image;
computing a virtual white point value for use in rendering synthetic content on the display device in association with the captured image, the virtual white point value based on the estimated white point;
determining a spectral reflectance of the synthetic content;
rendering the synthetic content based on the computed virtual white point value and the determined spectral reflectance of the synthetic content; and
displaying the rendered synthetic content on the display device in association with the captured image.

2. The method of claim 1 wherein the computed virtual white point value is selected from a finite group of white point values corresponding to real world illuminant sources.

3. The method of claim 1 further comprising:
obtaining at least one optical characteristic of a camera used to capture the image of the scene;
wherein the white point is further based on the at least one optical characteristic of the camera.

4. The method of claim 1 wherein the augmented reality display device is augmented reality goggles.

5. The method of claim 4 further comprising:
receiving an optical property of the augmented reality goggles; and
wherein the rendering of the synthetic content is further based on the optical property of the augmented reality goggles.

6. The method of claim 5 wherein the optical property of the augmented reality goggles comprises an optical transmissivity of the augmented reality goggles.

7. The method of claim 1 further comprising:
determining a difference between the white point of the captured image and a white point of a previously captured image that was previously displayed on the display device along with the synthetic content; and
wherein the rendering comprises transitioning the rendering of the synthetic content between a rendering based on the virtual white point of the previously rendered image and a rendering based on the virtual white point of the captured image in multiple, temporally-spaced increments.

8. The method of claim 7 wherein the transitioning comprises:
dividing a difference between the virtual white point of the previously rendered image and the virtual white point of the captured image into a plurality of incremental white points; and
successively rendering the synthetic content based on successive ones of the incremental virtual white points for successive periods of time until the virtual content has been rendered and displayed based on all of the successive incremental virtual white points.

9. The method of claim 1 further comprising:
determining a difference between the white point of the captured image and white point of a previously captured image that was previously displayed on the display device along with the synthetic content;
wherein the rendering comprises:
if the difference exceeds a threshold, transitioning the rendering of the synthetic content between a virtual white point based on the white point of the previously rendered image and a virtual white point based on the white point of the captured image in multiple, temporally spaced increments; and
if the difference does not exceed the threshold, transitioning the rendering of the synthetic content between the virtual white point based on the white point of the previously rendered image and the virtual white point based on the white point of the captured image in one increment.

10. An apparatus for displaying synthetic content in association with real-world content on a display device, the apparatus comprising:
a camera configured to capture an image;
a processor configured to:
receive the image from the camera;
estimate a white point of light illuminating a scene based on the image;
compute a virtual white point for use in rendering synthetic content on the display device in association with the image, the virtual white point based on the estimated white point;
determine a spectral reflectance of the synthetic content;
render the synthetic content based on the computed white point and the determined spectral reflectance of the synthetic content; and
send the rendered synthetic content to the display device for display in association with the image; and

the display device configured to:
display the image in association with the synthetic content; and
display the synthetic content in association with the image.

11. The apparatus of claim 10 wherein the processor is further configured to compute the virtual white point by selection from a finite group of white points corresponding to real-world illuminant sources.

12. The apparatus of claim 10 wherein the processor is further configured to:
obtain at least one optical characteristic of a camera; and
further base the estimated white point on the at least one optical characteristic of the camera.

13. The apparatus of claim 10 wherein the display device is goggles and the camera is mounted on the goggles in a forward-facing direction.

14. The apparatus of claim 13 wherein the processor is further configured to receive an optical property of the goggles, and wherein the rendering of the synthetic content is further based on the optical property of the goggles.

15. The apparatus of claim 14 wherein the optical property of the augmented reality goggles is optical transmissivity.

16. The apparatus of claim 10 wherein the processor is further configured to:
determine a difference between the white point of the image and a white point of a previous image that was previously displayed on the display device along with the synthetic content; and
wherein the rendering comprises transitioning the rendering of the synthetic content between a rendering based on the virtual white point of the previously rendered image and a rendering based on the virtual white point of the image in multiple, temporally-spaced increments.

17. The apparatus of claim 16 wherein the processor is further configured to affect the transitioning by:
dividing a difference between the virtual white point of the previously rendered image and the virtual white point of the image into a plurality of incremental white points; and
successively rendering the synthetic content based on successive ones of the incremental virtual white points for successive periods of time until the synthetic content has been rendered and displayed based on all of the incremental virtual white points.

18. The apparatus of claim 10 wherein the processor is further configured to;
determine a difference between the white point of the image and a white point of a previous image that was previously displayed on the display device along with the synthetic content;
determine if the difference between the white point of the image and the white point of the previous image exceeds a threshold;
if the difference exceeds the threshold, transition the rendering of the synthetic content between a virtual white point based on the white point of the previously rendered image and virtual white point based on the white point of the image in multiple, temporally spaced increments; and
if the difference does not exceed the threshold, transition the rendering of the synthetic content between the virtual white point based on the white point of the previously rendered image and the virtual white point based on the white point of the image in one increment.

Assignees

• Pcms Holdings Inc

Inventors

• Wyble David

Classifications

• G02B27/0172

  characterised by optical features · CPC title

• G06T19/006 Primary

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G02B2027/014

  comprising information/image processing systems · CPC title

• G06T15/506

  Illumination models · CPC title

• G02B2027/0138

  comprising image capture systems, e.g. camera · CPC title