Foveated rendering

Patent metadata

Field	Value
Publication number	US-11010956-B2
Application number	US-201615372589-A
Country	US
Kind code	B2
Filing date	Dec 8, 2016
Priority date	Dec 9, 2015
Publication date	May 18, 2021
Grant date	May 18, 2021

Abstract

Foveated rendering for rendering an image uses a ray tracing technique to process graphics data for a region of interest of the image, and a rasterisation technique is used to process graphics data for other regions of the image. A rendered image can be formed using the processed graphics data for the region of interest of the image and the processed graphics data for the other regions of the image. The region of interest may correspond to a foveal region of the image. Ray tracing naturally provides high detail and photo-realistic rendering, which human vision is particularly sensitive to in the foveal region; whereas rasterisation techniques are suited for providing temporal smoothing and anti-aliasing in a simple manner, and is therefore suited for use in the regions of the image that a user will see in the periphery of their vision.

First claim

The invention claimed is:

1. A processing system configured to render one or more images of a scene, the processing system comprising:
rendering logic configured to process graphics data to generate an initial image comprising pixel values representing an image of the scene;
region identification logic configured to identify one or more regions of the initial image;
ray tracing logic configured to perform ray tracing to determine ray traced data for the identified one or more regions of the initial image; and
update logic configured to update the initial image using the determined ray traced data for the identified one or more regions of the initial image by adding detail to the identified one or more regions of the initial image, to thereby determine an updated image to be outputted for display;
wherein the rendering logic and the ray tracing logic are configured to operate asynchronously; and
wherein the ray tracing logic, the region identification logic and the update logic are configured to operate at a first frame rate and the rendering logic is configured to operate at a second frame rate, said first frame rate being faster than said second frame rate.

2. The processing system of claim 1 wherein the rendering logic is configured to process the graphics data using a rasterisation technique to generate the initial image.

3. The processing system of claim 1 wherein the rendering logic is configured to process the graphics data using a ray tracing technique to generate the initial image.

4. The processing system of claim 1 wherein the initial image is a lower detail image than the updated image.

5. The processing system of claim 1 further comprising gaze tracking logic configured to determine one or more gaze positions for the initial image, wherein the region identification logic is configured to receive one or more indications of the one or more determined gaze positions, and to identify the one or more regions of the initial image based on the one or more determined gaze positions.

6. The processing system of claim 5 wherein the gaze tracking logic is configured to implement a predictive model to anticipate movements in gaze.

7. The processing system of claim 5 wherein one of the one or more identified regions of the initial image surrounds one of the one or more determined gaze positions, thereby representing a foveal region.

8. The processing system of claim 7 further comprising a camera pipeline which is configured to:
receive image data from a camera which is arranged to capture images of a user looking at a display on which a rendered image is to be displayed; and
process the received image data to generate a captured image;
wherein the gaze tracking logic is configured to analyse the captured image to determine the gaze position for the initial image.

9. The processing system of claim 8 wherein the ray tracing logic and the rasterisation logic are implemented on a graphics processing unit,
and wherein the camera pipeline and the graphics processing unit are implemented as part of a system on chip (SOC).

10. The processing system of claim 1 wherein the region identification logic is configured to analyse the initial image to determine one or more regions of high frequency, wherein the one or more determined regions of high frequency are one or more identified regions of the initial image.

11. The processing system of claim 1 further comprising time warping logic configured to apply an image warping process to the updated image before it is sent for display.

12. The processing system of claim 1 further comprising acceleration structure building logic configured to determine an acceleration structure representing the graphics data of geometry in a scene of which an image is to be rendered.

13. The processing system of claim 12 wherein the processing system is configured to render a plurality of images representing a sequence of frames, and wherein the acceleration structure building logic is configured to determine the acceleration structure for a current frame by updating the acceleration structure for the preceding frame.

14. The processing system of claim 1 wherein the processing system is arranged to be included in a virtual reality system or an augmented reality system.

15. A method of rendering one or more images of a scene at a processing system, the method comprising:
processing graphics data to generate an initial image comprising pixel values representing an image of the scene;
identifying one or more regions of the initial image;
performing ray tracing to determine ray traced data for the identified one or more regions of the initial image; and
updating the initial image using the determined ray traced data for the identified one or more regions of the initial image by adding detail to the identified one or more regions of the initial image, to thereby determine an updated image to be outputted for display;
wherein said performing ray tracing, said identifying one or more regions of the initial image and said updating the initial image are performed at a first frame rate, and wherein said processing graphics data to generate an initial image is performed at a second frame rate, wherein the first frame rate is faster than the second frame rate.

16. The method of claim 15 further comprising displaying an image based on the updated image.

17. The method of claim 15 wherein the processing graphics data to generate an initial image is performed at an asynchronous rate to the performing ray tracing to determine traced data.

18. A non-transitory computer readable storage medium having stored thereon a computer readable dataset description of an integrated circuit that, when processed in an integrated circuit manufacturing system, causes the integrated circuit manufacturing system to manufacture a processing system comprising:
rendering logic configured to process graphics data to generate an initial image comprising pixel values representing an image of a scene;
region identification logic configured to identify one or more regions of the initial image;
ray tracing logic configured to perform ray tracing to determine ray traced data for the identified one or more regions of the initial image; and
update logic configured to update the initial image using the determined ray traced data for the identified one or more regions of the initial image by adding detail to the identified one or more regions of the initial image, to thereby determine an updated image to be outputted for display;
wherein the rendering logic and the ray tracing logic are configured to operate asynchronously;
wherein the ray tracing logic, the region identification logic and the update logic are configured to operate at a first frame rate, and the rendering logic is configured to operate at a second frame rate, the first frame rate being faster than the second frame rate.

Assignees

• Imagination Tech Ltd

Inventors

• Blackmon Steven
• Peterson Luke T
• Ozdas Cuneyt
• Clohset Steven J

Classifications

• G06T15/06 Primary

  Ray-tracing · CPC title

• G06T2207/30041

  Eye; Retina; Ophthalmic · CPC title

• G06T2210/36

  Level of detail · CPC title

• G06F3/013

  Eye tracking input arrangements (G06F3/015 takes precedence) · CPC title

• G06T15/005

  General purpose rendering architectures · CPC title