Depth based foveated rendering for display systems

Patent metadata

Field	Value
Publication number	US-11644669-B2
Application number	US-201815927808-A
Country	US
Kind code	B2
Filing date	Mar 21, 2018
Priority date	Mar 22, 2017
Publication date	May 9, 2023
Grant date	May 9, 2023

Abstract

Methods and systems for depth-based foveated rendering in the display system are disclosed. The display system may be an augmented reality display system configured to provide virtual content on a plurality of depth planes using different wavefront divergence. Some embodiments include monitoring eye orientations of a user of a display system based on detected sensor information. A fixation point is determined based on the eye orientations, the fixation point representing a three-dimensional location with respect to a field of view. Location information of virtual objects to present is obtained, with the location information indicating three-dimensional positions of the virtual objects. Resolutions of at least one virtual object is adjusted based on a proximity of the at least one virtual object to the fixation point. The virtual objects are presented to a user by display system with the at least one virtual object being rendered according to the adjusted resolution.

First claim

What is claimed is:

1. A system comprising one or more processors and one or more computer storage media storing instructions that when executed by the one or more processors, cause the one or more processors to perform operations comprising:
monitoring, based on information detected via one or more sensors, eye movements of a user, wherein the user has a field of view comprising a plurality of zones, each zone representing a volume of space within the field of view in which virtual objects are configured for presentation by the display device, wherein a first zone encompasses a first range of depths within the field of view, and wherein a second zone encompasses a second, different, range of depths within the field of view;
determining, based on the eye movements, a zone of the plurality of zones occupied by a fixation point at which the user's eyes are fixating, wherein the fixation point is a three-dimensional location in the field of view of the user;
obtaining location information associated with one or more virtual objects to present within the field of view to the user, the location information indicating three-dimensional positions of the virtual objects;
adjusting resolutions of at least one virtual object based, at least in part, on a proximity of 1) a zone in which the at least one virtual object is configured for presentation to 2) the zone occupied by the fixation point; and
causing presentation, via a display device to the user, of the one or more virtual objects, the at least one virtual object being rendered according to the respective adjusted resolution.

2. The system of claim 1 , wherein adjusting resolution of a virtual object comprises reducing a presentation quality of the virtual object.

3. The system of claim 2 , wherein reducing a presentation quality of the virtual object comprises one or more of reducing a polygon count of the virtual object, adjusting primitives utilized to generate the virtual object, adjusting operations performed on the virtual object, adjusting texture information, adjusting color resolution or depth, and adjusting a number of rendering cycles or a frame rate.

4. The system of claim 1 , wherein the field of view corresponds to a display frustum of the display device.

5. The system of claim 1 ,
wherein the plurality of zones extend from an initial depth within the field of view to an end depth within the field of view.

6. The system of claim 5 , wherein adjusting resolutions of the at least one virtual object comprises reducing resolution from a resolution initially assigned to the at least one virtual object, wherein the proximity corresponds to a total number of zones separating the first zone from the second zone.

7. The system of claim 5 , wherein the zone in which the fixation point is located is assigned a maximum resolution, and wherein remaining zones are assigned resolutions respectively reduced from the maximum resolution according to one or more drop-off rates.

8. The system of claim 5 , wherein the operations further comprise:
providing a user retinal cone density profile,
wherein a drop-off rate on an individual depth plane substantially conforms to a drop-off rate of cone density in the user retinal cone density profile.

9. The system of claim 5 , wherein the zones have three-dimensional polygonal shapes.

10. The system of claim 5 , wherein the zones are concentric ellipsoids.

11. The system of claim 5 , wherein a shape of the zones varies with distance to the second zone.

12. The system of claim 1 , further comprising the display device, the display device comprising:
a plurality of stacked waveguides forming a display area and providing a view of an ambient environment through the display area, wherein at least some waveguides of the plurality of waveguides are configured to output light with different wavefront divergence that other waveguides.

13. The system of claim 1 , wherein, for a first virtual object to be presented further in depth from the fixation point and which is determined to be within a threshold angular distance of a gaze of the user, the operations further comprise:
adjusting the resolution of the first virtual object based on a proximity of the first virtual object to the fixation point; and
causing the first virtual object to be blurred during presentation of the first virtual object to the user.

14. The system of claim 13 , wherein causing the first virtual object to be blurred comprises:
performing a convolution of a Gaussian blur kernel with the first virtual object rendered according to the adjusted resolution; and
presenting, via the display device, the result of the convolution to the user.

15. The system of claim 1 , wherein the one or more sensors comprise one or more of infrared sensors, ultraviolet sensors, and visible wavelength light imaging devices configured to detect gaze directions of eyes of the user.

16. A display system comprising:
a display device configured to present virtual content to a user;
one or more processors; and
one or more computer storage media storing instructions that when executed by the system, cause the system to perform operations comprising:
monitoring information associated with eye movements of the user;
determining, based on the monitored information, a fixation point within a display frustum of the display device, the fixation point indicating a three-dimensional location being fixated upon by eyes of the user, wherein the display frustum comprises a plurality of zones, each zone representing a volume of space within the display frustum in which virtual content is configured for presentation by the display device, wherein a first zone encompasses a first range of depths within the display frustum, and wherein a second zone encompasses a second, different, range of depths within the display frustum; and
based on the determined fixation point, presenting virtual content at three-dimensional locations within the display frustum, the virtual content being adjusted in resolution based on a zone of the plurality of zones in which the virtual content is presented.

17. The system of claim 16 , wherein the operations further comprise:
providing, by the display device to an outside system comprising one or more processors, the determined fixation point; and
receiving, by the display device and from the outside system, rendered virtual content for presentation, the virtual content being rendered at the adjusted resolution.

18. The system of claim 16 , wherein adjusting resolution of particular virtual content based on a proximity from the fixation point comprises:
determining a reduction in a maximum resolution based on the proximity, the reduction being based on a drop-off rate; and
adjusting resolution of the particular virtual content based on the determined reduction.

19. The system of claim 16 , wherein the display device comprises a plurality of waveguides, each waveguide presenting virtual content associated with respective depths based on three-dimensional locations of the virtual content.

20. The system of claim 16 , wherein the display device presents virtual content based on foveated imaging with respect to the fixation point, the foveated imaging incorporating depth information associated with (1) the fixation point, and (2) virtual content.

21. A method comprising:
by a system of one or more processors,
monitoring, based on information detected via one or more sensors, eye orientations of a user of display device, wherein the user has a fi

Assignees

• Magic Leap Inc

Inventors

• Yeoh Ivan Li Chuen
• Edwin Lionel Ernest
• Samec Nicole Elizabeth
• Robaina Nastasja U
• Mathur Vaibhav
• Dalrymple Timothy Mark
• Schaefer Jason
• Carlisle Clinton
• Cheng Hui-Chuan
• Oh Chulwoo
• Premysler Philip
• Zhang Xiaoyang
• Carlson Adam C

Classifications

• H04N13/139

  Format conversion, e.g. of frame-rate or size · CPC title

• H04N13/324

  Colour aspects · CPC title

• G02B27/0172 Primary

  characterised by optical features · CPC title

• H04N13/398

  Synchronisation thereof; Control thereof · CPC title

• G06T15/00

  Three-dimensional [3D] image rendering · CPC title