Direct light compensation technique for augmented reality system

Patent metadata

Field	Value
Publication number	US-10628996-B2
Application number	US-201715460009-A
Country	US
Kind code	B2
Filing date	Mar 15, 2017
Priority date	Mar 15, 2016
Publication date	Apr 21, 2020
Grant date	Apr 21, 2020

Abstract

An augmented reality system and method of operating the same is provided. An end user is allowed to visualize direct light from a three-dimensional scene in an ambient environment, generating virtual image data from the point of the view of the end user. A region of spatial overlap between a real object in the three-dimensional scene and a virtual object in the virtual image data is determined. Color characteristics of the real object in the overlap region are determined. A perceived contrast between the real object and the virtual object in the overlap region is decreased based on the determined color characteristics of the real object in the overlap region. The virtual image data is displayed as a virtual image after the perceived contrast between the real object and the virtual object has been decreased that, along with the visualized direct light, creates a three-dimensional augmented scene.

First claim

What is claimed is:

1. A method of operating an augmented reality (AR) system, the method comprising:
allowing an end user to visualize direct light from a three-dimensional scene in an ambient environment;
capturing image data of the real three-dimensional scene with at least one camera affixed relative to a head of a user;
generating virtual image data from the point of the view of the end user, wherein each of the image data captured with the at least one camera and the virtual image data comprises a plurality of pixels, each pixel of which is defined by at least one value;
determining an overlap region of a spatial overlap between a real object in the three-dimensional scene and a virtual object in the virtual image data, wherein determining the overlap region between the real object and the virtual object comprising:
determining the spatial overlap between a corresponding real object in the image data and the virtual object in the virtual image data;

determining a color of the real object in the overlap region at least by determining a corresponding color of the corresponding real object in the image data for the overlap region;
displaying the virtual image data as a virtual image to the end user that, along with the visualized direct light, creates a three-dimensional augmented scene; and
completely obscuring the real object in the overlap region from the perspective of the end user based on the color of the real object in the overlap region, wherein
completely obscuring the real object in the overlap region comprises modifying the at least one value of a pixel of the virtual object in the overlap region,
modifying the at least one value of the pixel of the virtual object in the overlap region comprises reducing the at least one value of the pixel of the virtual object, and
the at least one value of the pixel of the virtual object is reduced by at least one corresponding value of a corresponding pixel of the corresponding real object.

2. The method of claim 1 , wherein completely obscuring the real object in the overlap region comprises:
generating interference data based on the color of the real object in the overlap region; and
displaying the interference data as an interference image over the overlap region to the user such that the interference image combines with the direct light from the real object to create a background for the virtual object in the overlap region.

3. The method of claim 2 , wherein the background in the overlap region has a decreased dynamic color range relative to the real object in the overlap region.

4. The method of claim 3 , wherein the background has a generally uniform color in the overlap region.

5. The method of claim 4 , wherein the generally uniform color is grey.

6. The method of claim 2 , wherein the interference image adds a first color to the real object in the overlap region.

7. The method of claim 1 , wherein completely obscuring the real object in the overlap region comprises modifying the virtual image data based on the color of the real object in the overlap region.

8. The method of claim 7 , wherein the modifying the virtual image data comprises subtracting the color from the virtual object in the overlap region.

9. The method of claim 1 , wherein the each of the plurality of pixels has three values defining a color vector.

10. The method of claim 9 , wherein the color vector comprises a red value, a green value, and a blue value.

11. The method of claim 1 , wherein the at least one value of the pixel of the virtual object has three first values defining a first color vector, and the at least one value of the corresponding pixel of the real object has three second values defining a second color vector.

12. The method of claim 1 , wherein
completely obscuring the real object in the overlap region comprises:
selecting at least one background value of a background pixel of a background for the virtual object in the overlap region;
generating interference data comprising a first plurality of pixels, each of which is defined by at least one second value, and
displaying the interference data as an interference image over the overlap region to the user such that the interference image combines with the direct light from the real object to create the background for the virtual object in the overlap region.

13. The method of claim 12 , wherein the each of the first plurality of pixels has three values defining a color vector.

14. The method of claim 12 , wherein the color vector comprises a red value, a green value, and a blue value.

15. The method of claim 12 , wherein completely obscuring the real object in the overlap region further comprises selecting at least one background value of a background pixel of the background for the virtual object in the overlap region, wherein at least one interference value of a first corresponding pixel of the interference data, when respectively added to the at least one corresponding value of the corresponding pixel of the corresponding real object in the image data in the overlap region, equals the at least one background value of the background pixel of the background for the virtual object in the overlap region.

16. The method of claim 12 , wherein all of corresponding values for corresponding pixels of the corresponding real object in the image data for the overlap region are respectively below the at least one background value of the background pixel of the background for the virtual object in the overlap region.

17. The method of claim 16 , wherein completely obscuring the real object in the overlap region further comprises:
determining that at least one first corresponding value for another corresponding pixel of the corresponding real object in the image data in the overlap region is below at least one first background value of another corresponding background pixel of the background for the virtual object in the overlap region; and
modifying the at least one value of another corresponding pixel of the virtual object in the overlap region.

Assignees

• Magic Leap Inc

Inventors

• Kass Michael

Classifications

• G06T2219/2012

  Colour editing, changing, or manipulating; Use of colour codes · CPC title

• G06T19/20

  Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts · CPC title

• G06T15/506 Primary

  Illumination models · CPC title

• G06T19/006

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G06T2215/16

  Using real world measurements to influence rendering · CPC title