Depth based foveated rendering for display systems

Patent metadata

Field	Value
Publication number	US-12189128-B2
Application number	US-202318297191-A
Country	US
Kind code	B2
Filing date	Apr 7, 2023
Priority date	Mar 22, 2017
Publication date	Jan 7, 2025
Grant date	Jan 7, 2025

Abstract

Methods and systems for depth-based foveated rendering in the display system are disclosed. The display system may be an augmented reality display system configured to provide virtual content on a plurality of depth planes using different wavefront divergence. Some embodiments include monitoring eye orientations of a user of a display system based on detected sensor information. A fixation point is determined based on the eye orientations, the fixation point representing a three-dimensional location with respect to a field of view. Location information of virtual objects to present is obtained, with the location information indicating three-dimensional positions of the virtual objects. Resolutions of at least one virtual object is adjusted based on a proximity of the at least one virtual object to the fixation point. The virtual objects are presented to a user by display system with the at least one virtual object being rendered according to the adjusted resolution.

First claim

What is claimed is:

1. A display system comprising:
one or more processors; and
one or more computer storage media storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising:
determining an amount of light reaching a retina of an eye of a user of the display system;
adjusting resolution of virtual content to be presented to the user based on the amount of light reaching the retina, wherein adjusting the resolution of the virtual content includes increasing a spatial resolution of the virtual content with increasing amounts of light reaching the retina; and
causing presentation, via a display device, of the virtual content to the user, wherein the virtual content is presented according to the adjusted resolution.

2. The display system of claim 1 , further comprising:
a frame configured to mount on a head of the user; and
the display device, including:
a light modulating system configured to output light to form images; and
one or more waveguides attached to the frame and configured to receive the light from the light modulating system and to output the light across a surface of the one or more waveguides.

3. The display system of claim 2 , wherein the one or more waveguides comprise:
one or more incoupling diffractive optical elements;
one or more outcoupling diffractive optical elements; and
one or more light distributing elements configured to direct light from the one or more incoupling diffractive optical elements to the one or more outcoupling diffractive optical elements.

4. The display system of claim 2 , wherein the one or more waveguides comprise a stack of waveguides, wherein at least some of the waveguides of the stack of waveguides provide different amounts of wavefront divergence than other waveguides of the stack of waveguides.

5. The display system of claim 2 , wherein determining the amount of light reaching the retina further comprises determining an amount of light that is output by the one or more waveguides to the eye of the user.

6. The display system of claim 1 , further comprising an outwardly-directed camera configured to measure an ambient illumination level, wherein determining the amount of light reaching the retina comprises measuring the ambient illumination level using the outwardly-directed camera.

7. The display system of claim 1 , wherein determining the amount of light reaching the retina comprises determining a size of a pupil of the eye of the user.

8. The display system of claim 1 , wherein the operations further comprise:
determining a user fixation point; and
obtaining location information of the virtual content, the location information indicating three-dimensional positions of the virtual content,
wherein adjusting resolution of the virtual content further comprises varying resolutions of the virtual content based on a proximity of the virtual content to the user fixation point.

9. The display system of claim 1 , wherein adjusting resolution of the virtual content further comprises decreasing resolution of the virtual content with increasing distance of the virtual content from the user fixation point.

10. The display system of claim 9 , wherein a trend of the decreasing resolution is substantially similar to a trend of decreasing cone density in the retina of the user.

11. The display system of claim 10 , wherein a zone of maximum resolution within a user field of view is centered on a point corresponding to a foveola of the eye of the user,
wherein the resolution of virtual content within a user field of view decreases for virtual content disposed outside of +/−20° of the point.

12. The display system of claim 9 , wherein decreasing resolution comprises decreasing polygon count at all levels of light reaching the retina.

13. The display system of claim 1 , wherein adjusting resolution further comprises setting the resolution at one of at least two resolution levels.

14. The display system of claim 1 , wherein adjusting resolution of the virtual content further comprises decreasing one or both of a color depth and a contrast ratio with decreasing amounts of light reaching the retina.

15. The display system of claim 1 , wherein the operations further comprise:
determining whether the amount of light corresponds to photopic, mesotopic, or scotopic illumination levels,
wherein adjusting resolution of the virtual content further comprises setting the resolutions based upon whether the illumination level corresponds to a photopic, mesotopic, or scotopic illumination level.

16. The display system of claim 1 , wherein adjusting resolution of the virtual content further comprises, at a scotopic illumination level, displaying the virtual content using a single color.

17. The display system of claim 1 , wherein adjusting resolution of the virtual content further comprises decreasing a contrast ratio of images forming the virtual content as the amount of light reaching the retina of the eye decreases.

18. The display system of claim 1 , wherein the operations further comprise:
providing virtual content in a plurality of component colors,
wherein adjusting resolution of the virtual content further comprises providing different resolutions for different component colors.

19. The display system of claim 1 , wherein the operations further comprise preventing rendering of virtual content located within an opposite peripheral region for each eye of the user, wherein the opposite peripheral region is on a side of the user opposite a side of the user on which the eye is disposed.

20. The display system of claim 1 , wherein the operations further comprise preventing rendering of virtual content corresponding to an optic nerve blind spot of the eye.

Assignees

• Magic Leap Inc

Inventors

• Yeoh Ivan Li Chuen
• Edwin Lionel Ernest
• Samec Nicole Elizabeth
• Robaina Nastasja U
• Mathur Vaibhav
• Dalrymple Timothy Mark
• Schaefer Jason
• Carlisle Clinton
• Cheng Hui-Chuan
• Oh Chulwoo
• Premysler Philip
• Zhang Xiaoyang
• Carlson Adam C

Classifications

• G02B2027/0134

  of stereoscopic type · CPC title

• G02B2027/0185

  Displaying image at variable distance · CPC title

• H04N2213/002

  Eyestrain reduction by processing stereoscopic signals or controlling stereoscopic devices · CPC title

• G06T15/00

  Three-dimensional [3D] image rendering · CPC title

• H04N13/398

  Synchronisation thereof; Control thereof · CPC title