Direct light compensation technique for augmented reality system

Patent metadata

Field	Value
Publication number	US-11257285-B2
Application number	US-202016812343-A
Country	US
Kind code	B2
Filing date	Mar 8, 2020
Priority date	Mar 15, 2016
Publication date	Feb 22, 2022
Grant date	Feb 22, 2022

Abstract

An augmented reality system and method of operating the same is provided. An end user is allowed to visualize direct light from a three-dimensional scene in an ambient environment, generating virtual image data from the point of the view of the end user. A region of spatial overlap between a real object in the three-dimensional scene and a virtual object in the virtual image data is determined. Color characteristics of the real object in the overlap region are determined. A perceived contrast between the real object and the virtual object in the overlap region is decreased based on the determined color characteristics of the real object in the overlap region. The virtual image data is displayed as a virtual image after the perceived contrast between the real object and the virtual object has been decreased that, along with the visualized direct light, creates a three-dimensional augmented scene.

First claim

What is claimed is:

1. A method of operating an augmented reality (AR) system, the method comprising:
warping first image data into warped image data based at least in part a perspective of a user and a position of the capturing device, wherein the first image data is captured by a capturing device from direct light in a three-dimensional scene;
determining an overlap region between a real object in the three-dimensional scene and a virtual object in virtual image data; and
presenting a three-dimensional augmented scene to the user at least by:
determining a characteristic of the real object in the overlap region;
identifying a pixel to be presented in the three-dimensional augmented scene to the user;
identifying real object data for displaying the characteristic of the real object at the pixel from the warped image data;
identifying virtual object data for displaying a corresponding portion of the virtual object at the pixel from the virtual image data;
determining pixel interference data for displaying the three-dimensional augmented scene at the pixel based at least in part upon the first data and the virtual object data; and
displaying, along with the direct light, the three-dimensional augmented scene to the user at least by adding the real object data, the pixel interference data, and the virtual object data for the pixel so that that the characteristic of the real object appears to be completely obscured by the virtual object from the perspective of the user;
wherein displaying the three-dimensional augmented scene comprises
generating the pixel interference data is based at least in part on the characteristic of the real object in the overlap region; and
displaying the pixel interference data as an interference image over the overlap region to the user such that the interference image combines with the direct light from the real object to create a background for the virtual object in the overlap region.

2. The method of claim 1 , wherein the background in the overlap region comprises a decreased dynamic color range relative to the real object in the overlap region.

3. The method of claim 2 , wherein the background has a generally uniform color in the overlap region.

4. The method of claim 3 , wherein the generally uniform color is grey.

5. The method of claim 1 , wherein the interference image adds a color to the real object in the overlap region.

6. The method of claim 1 , wherein displaying, along with the direct light, the three-dimensional augmented scene comprises modifying the virtual image data based at least in part on the characteristic of the real object in the overlap region.

7. The method of claim 6 , wherein the modifying the virtual image data comprises subtracting color data from the virtual object in the overlap region.

8. The method of claim 1 , further comprising:
capturing the first image data of the real three-dimensional scene with at least the capturing device that is affixed relative to a head of the user,
determining the overlap region at least by determining a spatial overlap between a corresponding real object in the first image data that has been captured and the virtual object in the virtual image data; and
determining the first contrast characteristic of the real object in the overlap region comprises determining a corresponding characteristic of the corresponding real object in the first image data in the overlap region.

9. The method of claim 8 , wherein each of the first image data and the virtual image data comprises a plurality of pixels, each of the plurality of pixels is defined by at least one value, and displaying, along with the direct light, the three-dimensional augmented scene comprises modifying the at least one value of a first pixel of the virtual object in the overlap region.

10. The method of claim 9 , wherein the each of the plurality of pixels has three values defining a color vector.

11. The method of claim 10 , wherein the color vector comprises a red value, a green value, and a blue value.

12. The method of claim 9 , wherein modifying the at least one value of the each of the plurality of pixels of the virtual object in the overlap region comprises reducing the at least one value of the first pixel of the virtual object.

13. The method of claim 12 , wherein the at least one value of the first pixel of the virtual object is reduced by at least one corresponding value of a corresponding pixel of the real object.

14. The method of claim 13 , wherein the at least one value of the first pixel of the virtual object has three first values defining a first color vector, and at least one corresponding value of the corresponding pixel of the real object has three second values defining a second color vector.

15. A method of operating an augmented reality (AR) system, the method comprising:
capturing first image data from direct light in a three-dimensional scene with at least a capturing device that is affixed relative to a head of an user;
warping first image data into warped image data based at least in part a perspective of a user and a position of the capturing device;
determining an overlap region between a real object in the three-dimensional scene and a virtual object in virtual image data; and
presenting a three-dimensional augmented scene to the user at least by:
determining a characteristic of the real object in the overlap region;
identifying a pixel to be presented in the three-dimensional augmented scene to the user;
identifying real object data for displaying the characteristic of the real object at the pixel from the warped image data;
identifying virtual object data for displaying a corresponding portion of the virtual object at the pixel from the virtual image data;
determining pixel interference data for displaying the three-dimensional augmented scene at the pixel based at least in part upon the first data and the virtual object data;

displaying, along with the direct light, the three-dimensional augmented scene to the user at least by adding the real object data, the pixel interference data, and the virtual object data for the pixel so that that the characteristic of the real object appears to be completely obscured by the virtual object from the perspective of the user;
determining the overlap region at least by determining a spatial overlap between a corresponding real object in the first image data that has been captured and the virtual object in the virtual image data; and
determining the first contrast characteristic of the real object in the overlap region comprises determining a corresponding characteristic of the corresponding real object in the first image data in the overlap region, wherein
each of the first image data and the virtual image data comprises a plurality of pixels, wherein each of the plurality of pixels is defined by at least one value; and
displaying, along with the direct light, the three-dimensional augmented scene comprises:
selecting at least one value of a first pixel of a background for the virtual object in the overlap region;
generating interference data comprising a first plurality of pixels, wherein each of the first plurality of pixels is defined by at least one value; and

displaying the interference data as an interference image over the overlap region to the user such that the interference image is combined with the direct light to create the background for the virtual object in the overlap region.

16. The method of claim 15 , wherein each of the first plurality of pixels has three values defining a color vector.

17. The method o

Assignees

• Magic Leap Inc

Inventors

• Kass Michael

Classifications

• G06T11/10 Primary

  Texturing; Colouring; Generation of textures or colours (retouching, inpainting or scratch removal G06T5/77) · CPC title

• G06T19/006

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G06T15/506 Primary

  Illumination models · CPC title

• G06T2219/2012

  Colour editing, changing, or manipulating; Use of colour codes · CPC title

• G06T2215/16

  Using real world measurements to influence rendering · CPC title