Apparatus and method for foveated rendering, bin comparison and TBIMR memory-backed storage for virtual reality implementations

Patent metadata

Field	Value
Publication number	US-10649524-B2
Application number	US-201715482694-A
Country	US
Kind code	B2
Filing date	Apr 7, 2017
Priority date	Apr 7, 2017
Publication date	May 12, 2020
Grant date	May 12, 2020

Abstract

One embodiment of a virtual reality apparatus comprises: a graphics processing engine comprising a plurality of graphics processing stages, the graphics processing engine to render a plurality of image frames for left and right displays of a head mounted display (HMD); and foveation control hardware logic to independently control two or more of the plurality of graphics processing stages based on feedback received from an eye tracking module of the HMD, the feedback indicating a foveated region selected based on a current or anticipated direction of a user's gaze, the foveation control hardware logic to cause the two or more of the graphics processing stages to process the foveated region differently than other regions of the image frames.

First claim

What is claimed is:

1. A virtual reality apparatus comprising:
a graphics processing engine comprising a plurality of graphics processing stages, the graphics processing engine to render a plurality of image frames for left and right displays of a head mounted display (HMD); and
foveation control hardware logic to independently control two or more of the plurality of graphics processing stages based on feedback received from an eye tracking module of the HMD, the feedback indicating a foveated region selected based on a current or anticipated direction of a user's gaze, the foveation control hardware logic to cause the two or more of the graphics processing stages to process the foveated region differently than other regions of the image frames,
wherein the foveation control hardware logic, based on the feedback indicating the foveated region, reduces a threshold for path termination for the foveated region to cause a shader to generate additional secondary rays for the foveated region and reduces a number of secondary rays for non-foveated regions.

2. The virtual reality apparatus as in claim 1 further comprising:
an eye tracking device in the HMD to track the current direction of the user's gaze.

3. The virtual reality apparatus as in claim 1 wherein the graphics processing stages to be controlled by the foveation control hardware logic comprises two or more of:
an input assembler to read index and vertex data for the image frames;
a vertex shader to perform shading operations on each vertex of the image frames;
a rasterizer to rasterize the vertex data to generate pixel data;
a pixel shader (PS) to perform shading operations on the pixel data; and
a frame buffer to store each image frame prior to being displayed on the HMD.

4. The virtual reality apparatus as in claim 3 the foveation control hardware logic is to control the pixel shader to use a reduced color gamut and/or color bit depth for pixels outside of the foveated region, reducing the color depth outside of this region.

5. The virtual reality apparatus as in claim 3 wherein the foveation control hardware logic is to control the rasterizer to use coarser rasterization for pixels outside of the foveated region than for pixels within the foveated region.

6. The virtual reality apparatus as in claim 3 wherein the foveation control hardware logic is to control a texture mapping unit to utilize a coarser level of detail (LOD) outside of the foveated region than within the foveated region.

7. The virtual reality apparatus as in claim 3 wherein the foveation control hardware logic is to cause different types of shaders to be used for the foveated region than regions outside the foveated region.

8. The virtual reality apparatus as in claim 1 wherein the graphics processing stages to be controlled by the foveation control hardware logic comprises two or more of:
ray generation hardware logic to generate a plurality of rays for each image frame;
ray traversal hardware logic to trace the rays through a bounding volume hierarchy;
intersection hardware logic to determine intersection points between the rays and one or more primitives; and
shader hardware logic to perform shading operations on the intersection points.

9. The virtual reality apparatus as in claim 8 wherein the foveation control hardware logic is to control a type of shader being applied to the foveated region by the shading hardware logic.

10. A virtual reality method comprising:
rendering a plurality of image frames for left and right displays of a head mounted display (HMD), the rendering performed by a graphics processing engine comprising a plurality of graphics processing stages;
tracking a current or anticipated direction of a user's gaze within the HMD to generate a feedback signal;
independently controlling two or more of the plurality of graphics processing stages based on the feedback signal, the feedback signal indicating a foveated region selected based on the current or anticipated direction of a user's gaze; and
causing the two or more of the graphics processing stages to process the foveated region differently than other regions of the image frames, wherein a threshold for path termination for the foveated region is reduced, based on the feedback indicating the foveated region, to cause a shader to generate additional secondary rays for the foveated region, and wherein a number of secondary rays for non-foveated regions is reduced.

11. The method as in claim 10 further comprising:
tracking the current direction of the user's gaze with an eye tracking device in the HMD.

12. The method as in claim 10 wherein the graphics processing stages to be controlled comprise two or more of:
an input assembler to read index and vertex data for the image frames;
a vertex shader to perform shading operations on each vertex of the image frames;
a rasterizer to rasterize the vertex data to generate pixel data;
a pixel shader (PS) to perform shading operations on the pixel data; and
a frame buffer to store each image frame prior to being displayed on the HMD.

13. The method as in claim 12 the pixel shader is controlled to use a reduced color gamut and/or color bit depth for pixels outside of the foveated region, reducing the color depth outside of this region.

14. The method as in claim 12 wherein the rasterizer is controlled to use coarser rasterization for pixels outside of the foveated region than for pixels within the foveated region.

15. The method as in claim 12 wherein a texture mapping unit is controlled to utilize a coarser level of detail (LOD) outside of the foveated region than within the foveated region.

16. The method as in claim 12 wherein different types of shaders are used for the foveated region than for regions outside the foveated region.

17. The method as in claim 10 wherein the graphics processing stages to be controlled comprise two or more of:
ray generation hardware logic to generate a plurality of rays for each image frame;
ray traversal hardware logic to trace the rays through a bounding volume hierarchy;
intersection hardware logic to determine intersection points between the rays and one or more primitives; and
shader hardware logic to perform shading operations on the intersection points.

18. The method as in claim 17 wherein a different type of shader is applied to the foveated region.

Assignees

• Intel Corp

Inventors

• Wald Ingo
• Insko Brent E
• Surti Prasoonkumar
• Lake Adam T
• Doyle Peter L
• Pohl Daniel

Classifications

• G06F3/013 Primary

  Eye tracking input arrangements (G06F3/015 takes precedence) · CPC title

• G06F3/011

  Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00) · CPC title

• G02B27/0093

  with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking · CPC title