Temporal supersampling for foveated rendering systems

Patent metadata

Field	Value
Publication number	US-11087437-B2
Application number	US-202016928527-A
Country	US
Kind code	B2
Filing date	Jul 14, 2020
Priority date	Jun 9, 2017
Publication date	Aug 10, 2021
Grant date	Aug 10, 2021

Abstract

Methods and systems are provided for using temporal supersampling to increase a displayed resolution associated with peripheral region of a foveated rendering view. A method for enabling reconstitution of higher resolution pixels from a low resolution sampling region for fragment data is provided. The method includes an operation for receiving a fragment from a rasterizer of a GPU and for applying temporal supersampling to the fragment with the low resolution sampling region over a plurality of prior frames to obtain a plurality of color values. The method further includes an operation for reconstituting a plurality of high resolution pixels in a buffer that is based on the plurality of color values obtained via the temporal supersampling. Moreover, the method includes an operation for sending the plurality of high resolution pixels for display.

First claim

What is claimed is:

1. A method, comprising:
tracking eye movement of a user, wherein attention of the user dynamically moves across a display;
processing a plurality of frames in a graphics pipeline when executing an application, wherein the attention of the user is directed to a foveal region of the display for the plurality of frames;
sampling at different locations of a low resolution pixel when processing each of the plurality of frames and storing a plurality of sampled color values in a frame buffer, wherein the low resolution pixel is located in a peripheral region that is outside the foveal region; and
reconstituting a plurality of high resolution pixels corresponding to the low resolution pixel by mapping sampled color values from the plurality of sampled color values for a frame and one or more prior frames to the plurality of high resolution pixels.

2. The method of claim 1 , further comprising:
mapping each of the different locations of the low resolution pixel to one of the plurality of high resolution pixels.

3. The method of claim 1 , further comprising:
processing a fragment of the frame corresponding to the low resolution pixel to determine a sampled color value for a corresponding location of the low resolution pixel.

4. The method of claim 1 , further comprising:
storing each of the plurality of sampled color values that are mapped to the plurality of high resolution pixels in a display buffer; and
sending from the display buffer the plurality of sampled color values that are mapped to the plurality of high resolution pixels for presentation on the display.

5. The method of claim 1 , wherein the sampling at different locations includes:
sampling the different locations of the low resolution pixel according to a pattern determined by jitter.

6. The method of claim 1 , further comprising:
sampling each location of the plurality of high resolution pixels in the foveal region when processing each of the plurality of frames, such that each of the plurality of frames includes high resolution pixel data in the foveal region.

7. The method of claim 1 , wherein the reconstituting the plurality of high resolution pixels includes:
blending of the plurality of sampled color values.

8. The method of claim 1 , wherein the plurality of high resolution pixels is associated with a native resolution of the display or a resolution that is greater than the native resolution of the display.

9. The method claim 1 , wherein the display is associated with a head mounted display (HMD).

10. A non-transitory computer-readable storage medium storing a computer program executable by a processor-based system, comprising:
program instructions for tracking eye movement of a user, wherein attention of the user dynamically moves across a display;
program instructions for processing a plurality of frames in a graphics pipeline when executing an application, wherein the attention of the user is directed to a foveal region of the display for the plurality of frames;
program instructions for sampling at different locations of a low resolution pixel when processing each of the plurality of frames and storing a plurality of sampled color values in a frame buffer, wherein the low resolution pixel is located in a peripheral region that is outside the foveal region; and
program instructions for reconstituting a plurality of high resolution pixels corresponding to the low resolution pixel by mapping sampled color values from the plurality of sampled color values for a frame and one or more prior frames to the plurality of high resolution pixels.

11. The non-transitory computer-readable storage medium of claim 10 , further comprising:
program instructions for mapping each of the different locations of the low resolution pixel to one of the plurality of high resolution pixels.

12. The non-transitory computer-readable storage medium of claim 10 , further comprising:
program instructions for processing a fragment of the frame corresponding to the low resolution pixel to determine a sampled color value for a corresponding location of the low resolution pixel.

13. The non-transitory computer-readable storage medium of claim 10 , further comprising:
program instructions for storing each of the plurality of sampled color values that are mapped to the plurality of high resolution pixels in a display buffer; and
program instructions for sending from the display buffer the plurality of sampled color values that are mapped to the plurality of high resolution pixels for presentation on the display.

14. The non-transitory computer-readable storage medium of claim 10 , wherein the program instructions for sampling at different locations includes:
program instructions for sampling the different locations of the low resolution pixel according to a pattern determined by jitter.

15. The non-transitory computer-readable storage medium of claim 10 , further comprising:
program instructions for sampling each location of the plurality of high resolution pixels in the foveal region when processing each of the plurality of frames, such that each of the plurality of frames includes high resolution pixel data in the foveal region.

16. A computer system comprising:
a processor;
a graphics system including a graphics pipeline, a frame buffer, and a display buffer; and
memory coupled to the processor and having stored therein instructions that, if executed by the computer system, cause the computer system to execute a method comprising:
tracking eye movement of a user, wherein attention of the user dynamically moves across a display;
processing a plurality of frames in the graphics pipeline when executing an application, wherein the attention of the user is directed to a foveal region of the display for the plurality of frames;
sampling at different locations of a low resolution pixel when processing each of the plurality of frames and storing a plurality of sampled color values in the frame buffer, wherein the low resolution pixel is located in a peripheral region that is outside the foveal region; and
reconstituting a plurality of high resolution pixels corresponding to the low resolution pixel by mapping sampled color values from the plurality of sampled color values for a frame and one or more prior frames to the plurality of high resolution pixels.

17. The computer system of claim 16 , the method further comprising:
mapping each of the different locations of the low resolution pixel to one of the plurality of high resolution pixels.

18. The computer system of claim 16 , the method further comprising:
processing a fragment of the frame corresponding to the low resolution pixel to determine a sampled color value for a corresponding location of the low resolution pixel.

19. The computer system of claim 16 , the method further comprising:
storing each of the plurality of sampled color values that are mapped to the plurality of high resolution pixels in a display buffer; and
sending from the display buffer the plurality of sampled color values that are mapped to the plurality of high resolution pixels for presentation on the display.

20. The computer system of claim 16 , wherein in the method the sampling at different locations includes:
sampling the different locations of the low resolution pixel according to a pattern determined by jitter.

Assignees

• Sony Interactive Entertainment Inc

Inventors

• Young Andrew
• Ho Chris
• Stafford Jeffrey Roger

Classifications

• G06F3/013 Primary

  Eye tracking input arrangements (G06F3/015 takes precedence) · CPC title

• G09G2350/00

  Solving problems of bandwidth in display systems · CPC title

• G06F3/012

  Head tracking input arrangements · CPC title

• G06T3/40 Primary

  Scaling of whole images or parts thereof, e.g. expanding or contracting · CPC title

• G06T3/4053

  based on super-resolution, i.e. the output image resolution being higher than the sensor resolution · CPC title