Real-time user adaptive foveated rendering

Patent metadata

Field	Value
Publication number	US-10192528-B2
Application number	US-201615086645-A
Country	US
Kind code	B2
Filing date	Mar 31, 2016
Priority date	Mar 31, 2016
Publication date	Jan 29, 2019
Grant date	Jan 29, 2019

Abstract

Foveated rendering based on user gaze tracking may be adjusted to account for the realities of human vision. Gaze tracking error and state parameters may be determined from gaze tracking data representing a user's gaze with respect to one or more images presented to a user. Adjusted foveation data representing an adjusted size and/or shape of one or more regions of interest in one or more images to be subsequently presented to a user may be generated based on the one or more gaze tracking error or state parameters. Foveated image data representing one or more foveated images may be generated with the adjusted foveation data. The foveated images are characterized by level of detail within the one or more regions of interest and lower level of detail outside the one or more regions of interest. The foveated images may then be presented to the user.

First claim

The invention claimed is:

1. A graphics processing method comprising:
obtaining gaze tracking data representing a user's gaze with respect to one or more images presented to the user;
determining one or more gaze tracking parameters from gaze tracking data, wherein the gaze tracking parameters include one or more gaze tracking error parameters, wherein determining the one or more gaze tracking error parameters from the gaze tracking data includes determining whether the user's eye is moving in smooth pursuit;
generating adjusted foveation data representing an adjusted size and/or shape of one or more regions of interest in one or more images to be subsequently presented to the user based on the one or more gaze tracking parameters including the one or more gaze tracking error parameters;
generating foveated image data representing one or more foveated images using the adjusted foveation data, wherein the one or more foveated images are characterized by level of detail within the one or more regions of interest and lower level of detail outside the one or more regions of interest; and
presenting the one or more foveated images to the user.

2. The method of claim 1 , wherein determining one or more gaze tracking error parameters from the gaze tracking data includes determining a rate of rotation of a user's eye with respect to one or more axes.

3. The method of claim 1 , wherein determining one or more gaze tracking error parameters from the gaze tracking data includes determining a rate of rotation of a user's eye with respect to one or more axes and determining an error in a fixation of the user's eye on a region of interest of the one or more regions of interest.

4. The method of claim 1 , wherein determining one or more gaze tracking error parameters from the gaze tracking data includes determining a confidence interval regarding the current and/or predicted future gaze position.

5. The method of claim 1 , further comprising wherein the gaze tracking parameters further include gaze tracking state parameters.

6. The method of claim 5 , wherein determining one or more gaze tracking state parameters from the gaze tracking data includes determining the metrics of a blink of a user.

7. The method of claim 5 , wherein determining one or more gaze tracking state parameters from the gaze tracking data includes determining the metrics of a saccade of a user.

8. The method of claim 5 , wherein determining one or more gaze tracking state parameters from the gaze tracking data includes determining a transition in the gaze direction of a user as a result of a change in the depth of field of the presented image.

9. The method of claim 5 , wherein determining one or more gaze tracking state parameters from the gaze tracking data includes determining whether a user is color blind.

10. The method of claim 5 , wherein determining one or more gaze tracking state parameters from the gaze tracking data includes determining a rate of rotation of a user's eye with respect to one or more axes and determining the level of gaze stability of the user's eye.

11. The method of claim 5 , wherein determining one or more gaze tracking state parameters from the gaze tracking data includes determining a rate of rotation of a user's eye with respect to one or more axes and determining whether the user's eye movement is a precursor to the user's head movement.

12. The method of claim 1 , wherein the adjusted foveation data includes data characterizing a particular region of interest of the one or more regions of interest, wherein the data characterizing the particular region of interest includes a foveation region and a transition region, wherein the foveation region is characterized by a higher level of detail than the transition region and the transition region is characterized by a higher level of detail than regions of a corresponding image outside the particular region of interest.

13. The method of claim 11 , wherein generating adjusted foveation data representing an adjusted size and/or shape of one or more regions of interest includes adjusting a size of the transition region.

14. The method of claim 1 , wherein the adjusted foveation data includes data used to determine geometric density in the one or more regions of interest.

15. The method of claim 1 , wherein the adjusted foveation data includes data used to determine pixel resolution in the one or more regions of interest.

16. The method of claim 1 , wherein the adjusted foveation data includes data used to determine pixel shading quality in the one or more regions of interest.

17. The method of claim 1 , further comprising creating a standard 2D compliant image from the foveated image data for presentation on additional displays.

18. A system, comprising:
a processor;
a memory; and
computer-readable instructions embodied in the memory, the computer-readable instructions being configured to implement a graphics processing method when executed, the graphics processing method comprising:
obtaining gaze tracking data representing a user's gaze with respect to one or more images presented to the user;
determining one or more gaze tracking parameters from gaze tracking data, wherein the gaze tracking parameters include at least one gaze tracking error parameter or at least one gaze tracking state parameter, wherein determining the one or more gaze tracking error parameters from the gaze tracking data includes determining whether the user's eye is moving in smooth pursuit;
generating adjusted foveation data representing an adjusted size and/or shape of one or more regions of interest in one or more images to be subsequently presented to the user based on the one or more gaze tracking parameters;
generating foveated image data representing one or more foveated images using the adjusted foveation data, wherein the one or more foveated images are characterized by level of detail within the one or more regions of interest and lower level of detail outside the one or more regions of interest; and
presenting the one or more foveated images to the user.

19. A non-transitory computer-readable medium having computer-readable instructions embodied therein, the computer-readable instructions being configured to implement a graphics processing method when executed, the graphics processing method comprising:
obtaining gaze tracking data representing a user's gaze with respect to one or more images presented to the user;
determining one or more gaze tracking parameters from gaze tracking data, wherein the gaze tracking parameters include at least one gaze tracking error parameter or at least one gaze tracking state parameter, wherein determining the one or more gaze tracking error parameters from the gaze tracking data includes determining whether the user's eye is moving in smooth pursuit;
generating adjusted foveation data representing an adjusted size and/or shape of one or more regions of interest in one or more images to be subsequently presented to the user based on the one or more gaze tracking error parameters;
generating foveated image data representing one or more foveated images using the adjusted foveation data, wherein the one or more foveated images are characterized by level of detail within the one or more regions of interest and lower level of detail outside the one or more regions of interest; and
presenting the one or more foveated images to the user.

Assignees

• Sony Computer Entertainment Inc
• Sony Interactive Entertainment Inc

Inventors

• Young Andrew
• Stafford Jeffrey Roger

Classifications

• G09G2340/0407

  Resolution change, inclusive of the use of different resolutions for different screen areas · CPC title

• G09G5/391 Primary

  Resolution modifying circuits, e.g. variable screen formats · CPC title

• G02B27/017

  Head mounted · CPC title

• G06F3/013 Primary

  Eye tracking input arrangements (G06F3/015 takes precedence) · CPC title

• G06F1/3265

  Power saving in display device · CPC title