Adaptive vehicle augmented reality display using stereographic imagery

Patent metadata

Field	Value
Publication number	US-11935197-B2
Application number	US-202117175456-A
Country	US
Kind code	B2
Filing date	Feb 12, 2021
Priority date	Sep 23, 2016
Publication date	Mar 19, 2024
Grant date	Mar 19, 2024

Abstract

An AR system that leverages a pre-generated 3D model of the world to improve rendering of 3D graphics content for AR views of a scene, for example an AR view of the world in front of a moving vehicle. By leveraging the pre-generated 3D model, the AR system may use a variety of techniques to enhance the rendering capabilities of the system. The AR system may obtain pre-generated 3D data (e.g., 3D tiles) from a remote source (e.g., cloud-based storage), and may use this pre-generated 3D data (e.g., a combination of 3D mesh, textures, and other geometry information) to augment local data (e.g., a point cloud of data collected by vehicle sensors) to determine much more information about a scene, including information about occluded or distant regions of the scene, than is available from the local data.

First claim

What is claimed is:

1. A system, comprising:
a display device; and
a controller comprising:
one or more processors; and
a memory storing instructions that, when executed on or across the one or more processors, cause the one or more processors to:
obtain sensor data for an environment of a real-world scene captured by one or more sensors;
generate virtual content for the environment based at least on the sensor data and three-dimensional (3D) mesh data for the environment;
determine lighting information for a region of the environment on which the virtual content is to be projected, the lighting information indicating a location or orientation of a light source relative to one or more surfaces of the region of the environment;
modify an appearance of the virtual content based at least on the determined lighting information of the region by moving a portion of the virtual content that is to be projected on or near the surface to another location; and
provide the virtual content to the display device.

2. The system of claim 1 , wherein the memory further comprises instructions that, when executed on or across the one or more processors, cause the one or more processors to:
determine a surface normal of a surface within the region of the environment;
determine a direction of light from the light source; and
determine that the surface reflects the light in the direction of a viewer based at least on the direction of the light and the surface normal.

3. The system of claim 2 , wherein the 3D mesh data comprises information indicating the surface normal.

4. The system of claim 2 , wherein to determine that the surface reflects the light in the direction of the viewer, the memory further comprises instructions that, when executed on or across the one or more processors, cause the one or more processors to:
determine a current position and direction of a vehicle comprising the display device and the controller.

5. The system of claim 1 , wherein the memory further comprises instructions that, when executed on or across the one or more processors, cause the one or more processors to:
modify the appearance of the virtual content by changing an intensity or a size of the virtual content based, at least in part, on the determined lighting information of the region.

6. The system of claim 1 , wherein the instructions, when executed on or across the one or more processors, cause the one or more processors to:
determine one or more colors of the region of the environment on which the virtual content is to be projected wherein the modification of the appearance of the virtual content is further based on the determined one or more colors of the region on which the virtual content is to be projected.

7. The system of claim 1 , wherein the display device is incorporated into a windshield of a vehicle.

8. A method, comprising:
performing, with one or more computing devices:
obtaining sensor data for an environment of a real-world scene captured by one or more sensors;
generating virtual content for the environment based at least on the sensor data and three-dimensional (3D) mesh data for the environment;
determining lighting information for a region of the environment on which the virtual content is to be projected, the lighting information indicating a location or orientation of a light source relative to one or more surfaces of the region of the environment;
modifying an appearance of the virtual content based at least on the determined lighting information of the region by moving a portion of the virtual content that is to be projected on or near the surface to another location; and
providing the virtual content to a display device.

9. The method of claim 8 , further comprising:
determining a surface normal of a surface within the region of the environment;
determining a direction of light from the light source; and
determining that the surface reflects the light in the direction of a viewer based at least on the direction of the light and the surface normal.

10. The method of claim 9 , wherein the 3D mesh data comprises information indicating the surface normal.

11. The method of claim 9 , wherein determining that the surface reflects the light in the direction of the viewer comprises:
determining a current position and direction of a vehicle comprising the display device and the controller.

12. The method of claim 8 , wherein modifying the appearance of the virtual content comprises:
modifying an intensity or a size of the virtual content based, at least in part, on the determined lighting information of the region.

13. The method of claim 8 , wherein modifying the appearance of the virtual content comprises:
modifying the appearance of the virtual content based, at least in part, on one or more colors of the region determined according to the sensor data.

14. The method of claim 8 , wherein the display device is incorporated into a windshield of a vehicle.

15. One or more non-transitory, computer-readable storage media storing instructions that, when executed on or across one or more processors, cause the one or more processors to:
obtain sensor data for an environment of a real-world scene captured by one or more sensors;
generate virtual content for the environment based at least on the sensor data and three-dimensional (3D) mesh data for the environment;
determine lighting information indicating for a region of the environment on which the virtual content is to be projected, the lighting information a location or orientation of a light source relative to one or more surfaces of the region of the environment;
modify an appearance of the virtual content based at least on the determined lighting information of the region by moving a portion of the virtual content that is to be projected on or near the surface to another location; and
provide the virtual content to a display device.

16. The one or more non-transitory, computer-readable storage media of claim 15 , wherein the memory further comprises instructions that, when executed on or across the one or more processors, cause the one or more processors to:
determine a surface normal of a surface within the region of the environment;
determine a direction of light from the light source;
determine that the surface reflects the light in the direction of a viewer based at least on the direction of the light and the surface normal.

17. The one or more non-transitory, computer-readable storage media of claim 16 , wherein the 3D mesh data comprises information indicating the surface normal.

18. The one or more non-transitory, computer-readable storage media of claim 17 , wherein to determine that the surface reflects the light in the direction of the viewer, the memory further comprises instructions that, when executed on or across the one or more processors, cause the one or more processors to:
determine a current position and direction of a vehicle comprising the display device and the controller.

19. The one or more non-transitory, computer-readable storage media of claim 15 , wherein to modify the appearance of the virtual content the instructions, when executed on or across the one or more processors, cause the one or more processors to:
modify an intensity or a size of the virtual content based, at least in part, on the determined lighting information of the region.

20. The one or more non-transitory, computer-readable storage media of claim 15 , wherein to modify the appearance of the virtual content the instructions, w

Assignees

• Apple Inc

Inventors

• Piemonte Patrick S
• De Rocha Rosario Daniel
• Gosnell Jason D
• Meier Peter

Classifications

• G06T19/006 Primary

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G01C21/26

  specially adapted for navigation in a road network · CPC title

• G01C21/365

  Guidance using head up displays or projectors, e.g. virtual vehicles or arrows projected on the windscreen or on the road itself · CPC title

• G02B27/017

  Head mounted · CPC title

• G06T15/205

  Image-based rendering · CPC title