Introducing real-time lighting effects to illuminate real-world physical objects in see-through augmented reality displays

Patent metadata

Field	Value
Publication number	US-11087529-B2
Application number	US-201916586503-A
Country	US
Kind code	B2
Filing date	Sep 27, 2019
Priority date	Sep 27, 2019
Publication date	Aug 10, 2021
Grant date	Aug 10, 2021

Abstract

Embodiments provide for the rendering of illumination effects on real-world objects in augmented reality systems. An example method generally includes overlaying a shader on the augmented reality display. The shader generally corresponds to a three-dimensional geometry of an environment in which the augmented reality display is operating, and the shader generally comprises a plurality of vertices forming a plurality of polygons. A computer-generated lighting source is introduced into the augmented reality display. One or more polygons of the shader are illuminated based on the computer-generated lighting source, thereby illuminating one or more real-world objects in the environment with direct lighting from the computer-generated lighting source and reflected and refracted lighting from surfaces in the environment.

First claim

What is claimed is:

1. A method for rendering lighting effects in an augmented reality display, comprising:
overlaying, on an augmented reality display, a shader corresponding to a three-dimensional geometry of an environment in which the augmented reality display is operating, wherein the shader comprises a plurality of vertices forming a plurality of polygons, a flat texture on which lighting effects can be rendered, and a plurality of vectors, each vector defining a light reflectivity value corresponding to a respective surface in the environment in which the augmented reality display is operating;
introducing a computer-generated lighting source into the augmented reality display, wherein the computer-generated lighting source comprises a light source that is absent from the environment in which the augmented reality display is operating and is rendered as an object overlaid in the environment in which the augmented reality display is operating; and
illuminating one or more polygons of the shader, thereby illuminating, in the augmented reality display, one or more real-world objects in the environment with direct lighting from the computer-generated lighting source and reflected and refracted lighting from surfaces in the environment wherein illuminating the one or more polygons comprises generating an illumination effect for at least one of the one or more real-world objects in the environment in which the augmented reality display is operating such that illumination effects against objects with higher reflectivity surfaces appear brighter than illuminations against objects with lower reflectivity surfaces.

2. The method of claim 1 , wherein the vertices of the shader have a color corresponding to a color of a display screen of the augmented reality display such that the shader, when unilluminated by the computer-generated lighting source, appears transparent.

3. The method of claim 1 , further comprising:
capturing the three-dimensional geometry of the environment in which the augmented reality display is operating based on a scan of the environment in which the augmented reality display is operating; and
generating the shader based on the captured three-dimensional geometry of the environment, wherein the vertices in the shader correspond to edges of surfaces detected in the three-dimensional geometry of the environment.

4. The method of claim 3 , further comprising:
measuring reflectivity of the surfaces in the environment in which the augmented reality display is operating, wherein the plurality of vectors defining light reflectivity values corresponding to a respective surface in the environment are based on the measured reflectivity of the respective surface.

5. The method of claim 4 , wherein illuminating the one or more real-world objects comprises decreasing values of one or more luminance channels for illuminated objects and increasing values of the one or more luminance channels for unilluminated objects such that illuminations against objects are perceived consistently through a tinted see-through optical display.

6. The method of claim 3 , wherein generating the shader based on the captured three-dimensional geometry of the environment comprises transmitting one or more images of the environment to a remote system via a low latency, high bandwidth network connection.

7. The method of claim 1 , wherein the three-dimensional geometry of the environment comprises a predefined three-dimensional model of the environment generated according to a known architectural layout of the environment and reflectivity of surfaces in the environment.

8. The method of claim 1 , wherein the vertices of the shader are colored to compensate for color vision deficiencies of a user.

9. The method of claim 1 , further comprising:
updating the three-dimensional geometry of the environment based on a detected change in the environment.

10. The method of claim 9 , wherein updating the three-dimensional geometry of the environment comprises:
measuring reflectivity of one or more changed surfaces identified in the detected change in the environment; and
updating one or more vectors in the shader based on the measured reflectivity of the one or more changed surfaces.

11. A system, comprising:
a processor; and
a memory having instructions stored thereon which, when executed by the processor, performs an operation for rendering lighting effects in an augmented reality display, the operation comprising:
overlaying, on an augmented reality display, a shader corresponding to a three-dimensional geometry of an environment in which the augmented reality display is operating, wherein the shader comprises a plurality of vertices forming a plurality of polygons, a flat texture on which lighting effects can be rendered, and a plurality of vectors, each vector defining a light reflectivity value corresponding to a respective surface in the environment in which the augmented reality display is operating;
introducing a computer-generated lighting source into the augmented reality display, wherein the computer-generated lighting source comprises a light source that is absent from the environment in which the augmented reality display is operating and is rendered as an object overlaid in the environment in which the augmented reality display is operating; and
illuminating one or more polygons of the shader, thereby illuminating, in the augmented reality display, one or more real-world objects in the environment with direct lighting from the computer-generated lighting source and reflected and refracted lighting from surfaces in the environment wherein illuminating the one or more polygons comprises generating an illumination effect for at least one of the one or more real-world objects in the environment in which the augmented reality display is operating such that illumination effects against objects with higher reflectivity surfaces appear brighter than illuminations against objects with lower reflectivity surfaces.

12. The system of claim 11 , wherein the vertices of the shader have a color corresponding to a color of a display screen of the augmented reality display such that the shader, when unilluminated by the computer-generated lighting source, appears transparent.

13. The system of claim 11 , wherein the operation further comprises:
capturing the three-dimensional geometry of the environment in which the augmented reality display is operating based on a scan of the environment in which the augmented reality display is operating;
measuring reflectivity of the surfaces in the environment in which the augmented reality display is operating; and
generating the shader based on the captured three-dimensional geometry of the environment and the measured reflectivity of the surfaces in the environment, wherein the vertices in the shader correspond to edges of surfaces detected in the three-dimensional geometry of the environment and the plurality of vectors defining light reflectivity values corresponding to a respective surface in the environment are based on the measured reflectivity of the surfaces in the environment.

14. The system of claim 13 , wherein illuminating the one or more real-world objects comprises decreasing values of one or more luminance channels for illuminated objects and increasing values of the one or more luminance channels for unilluminated objects such that illuminations against objects are perceived consistently through a tinted see-through optical display.

15. The system of claim 13 , wherein generating the shader based on the captured three-dimensional geometry of the environment comprises transmitting on

Assignees

• Disney Entpr Inc

Inventors

• Yeung Jason A
• Mitchell Kenneth J
• Panec Timothy M
• Baumbach Elliott H
• Drake Corey D

Classifications

• G02B2027/014

  comprising information/image processing systems · CPC title

• G02B27/0172

  characterised by optical features · CPC title

• G06T19/006

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G02B2027/0138

  comprising image capture systems, e.g. camera · CPC title

• G06T15/005

  General purpose rendering architectures · CPC title