Apparatus and method for foveated rendering, bin comparison and TBIMR memory-backed storage for virtual reality implementations

Patent metadata

Field	Value
Publication number	US-11269409-B2
Application number	US-202016867248-A
Country	US
Kind code	B2
Filing date	May 5, 2020
Priority date	Apr 7, 2017
Publication date	Mar 8, 2022
Grant date	Mar 8, 2022

Abstract

One embodiment of a virtual reality apparatus comprises: a graphics processing engine comprising a plurality of graphics processing stages, the graphics processing engine to render a plurality of image frames for left and right displays of a head mounted display (HMD); and foveation control hardware logic to independently control two or more of the plurality of graphics processing stages based on feedback received from an eye tracking module of the HMD, the feedback indicating a foveated region selected based on a current or anticipated direction of a user's gaze, the foveation control hardware logic to cause the two or more of the graphics processing stages to process the foveated region differently than other regions of the image frames.

First claim

What is claimed is:

1. An apparatus comprising:
a graphics processing engine comprising a plurality of graphics processing stages, the graphics processing engine to render a plurality of image frames for left and right displays of a head mounted display (HMD); and
control hardware logic to control the plurality of graphics processing stages based on feedback received from an eye tracking module of the HMD, the feedback indicating a foveated region selected based on a direction of a user's gaze, the control hardware logic to cause the plurality of graphics processing stages to process the foveated region differently than one or more non-foveated regions of the image frames,
the control hardware logic to cause a shader to reduce a number of secondary rays for the one or more non-foveated regions and to adjust a threshold for path termination utilized by the shader to generate the secondary rays based on the feedback indicating the foveated region.

2. The apparatus of claim 1 , wherein the control hardware logic is to further cause the shader to increase a number of secondary rays for the foveated region.

3. The apparatus of claim 1 , wherein the control hardware logic is to further reduce a bit size for color depth of the one or more non-foveated regions.

4. The apparatus of claim 1 , wherein the control hardware logic is to use, for the foveated region, a first shader selected from a set of shaders including a vertex shader, a geometry shader, and a pixel shader, and wherein the control hardware logic is to use, for the one or more non-foveated regions, a second shader selected from the set of shaders and different from the first shader.

5. The apparatus of claim 1 , wherein the control hardware logic is to further cause a coarser rasterization for pixels within the one or more non-foveated regions than for pixels within the foveated region.

6. The apparatus of claim 1 , wherein the control hardware logic is to further cause a texture mapping unit to utilize a coarser level of detail (LOD) within the one or more non-foveated regions than within the foveated region.

7. The apparatus of claim 1 , wherein the plurality of graphics processing stages comprises two or more of:
ray generation hardware logic to generate a plurality of rays for each image frame;
ray traversal hardware logic to trace the rays through a bounding volume hierarchy;
intersection hardware logic to determine intersection points between the rays and one or more primitives; and
shader hardware logic to perform shading operations on the intersection points.

8. The apparatus of claim 1 , wherein the control hardware logic is to further cause a greater level of compression for the one or more non-foveated regions than the foveated region.

9. The apparatus of claim 1 , wherein the control hardware logic is to further cause reduction of a sample count for the one or more non-foveated regions.

10. A method comprising:
rendering a plurality of image frames for left and right displays of a head mounted display (HMD) through a plurality of graphics processing stages;
controlling the plurality of graphics processing stages based on feedback received from an eye tracking module of the HMD, the feedback indicating a foveated region selected based on a direction of a user's gaze; and
causing the plurality of graphics processing stages to process the foveated region differently than one or more non-foveated regions of the image frames, wherein the processing comprises causing a shader to reduce a number of secondary rays for the one or more non-foveated regions and to adjust a threshold for path termination utilized by the shader to generate the secondary rays based on the feedback indicating the foveated region.

11. The method of claim 10 , wherein the processing further causes the shader to increase a number of secondary rays for the foveated region.

12. The method of claim 10 , wherein the processing further reduces a bit size for color depth of the one or more non-foveated regions.

13. The method of claim 10 , wherein the processing further comprises for the foveated region, using a first shader selected from a set of shaders including a vertex shader, a geometry shader, and a pixel shader, and for the one or more non-foveated regions, using a second shader selected from the set of shaders and different from the first shader.

14. A non-transitory computer-readable storage medium storing instructions that when executed by a processor of a computing system, are capable of causing the computing system to perform:
rendering a plurality of image frames for left and right displays of a head mounted display (HMD) through a plurality of graphics processing stages;
controlling the plurality of graphics processing stages based on feedback received from an eye tracking module of the HMD, the feedback indicating a foveated region selected based on a direction of a user's gaze; and
causing the plurality of graphics processing stages to process the foveated region differently than one or more non-foveated regions of the image frames, wherein the processing comprises causing a shader to reduce a number of secondary rays for the one or more non-foveated regions and to adjust a threshold for path termination utilized by the shader to generate the secondary rays based on the feedback indicating the foveated region.

15. The non-transitory computer-readable storage medium of claim 14 , wherein the processing further causes the shader to increase a number of secondary rays for the foveated region.

16. The non-transitory computer-readable storage medium of claim 14 , wherein the processing further reduces a bit size for color depth of the one or more non-foveated regions.

17. The non-transitory computer-readable storage medium of claim 14 , wherein the processing further comprises for the foveated region, using a first shader selected from a set of shaders including a vertex shader, a geometry shader, and a pixel shader, and for the non-foveated regions, using a second shader selected from the set of shaders and different from the first shader.

18. The non-transitory computer-readable storage medium of claim 14 , wherein the processing further uses a coarser rasterization for pixels within the one or more non-foveated regions than for pixels within the foveated region.

19. The non-transitory computer-readable storage medium of claim 14 , wherein the processing further causing a texture mapping unit to utilize a coarser level of detail (LOD) within the one or more non-foveated regions than within the foveated region.

Assignees

• Intel Corp

Inventors

• Wald Ingo
• Insko Brent E
• Surti Prasoonkumar
• Lake Adam T
• Doyle Peter L
• Pohl Daniel

Classifications

• G06F3/011

  Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00) · CPC title

• G02B27/0093

  with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking · CPC title

• G06F3/013 Primary

  Eye tracking input arrangements (G06F3/015 takes precedence) · CPC title