Optimized deferred lighting in a foveated rendering system

Patent metadata

Field	Value
Publication number	US-10515466-B2
Application number	US-201715727515-A
Country	US
Kind code	B2
Filing date	Oct 6, 2017
Priority date	Jun 9, 2017
Publication date	Dec 24, 2019
Grant date	Dec 24, 2019

Abstract

A method for implementing a graphics pipeline. The method includes determining a plurality of light sources affecting a virtual scene. Geometries of objects of an image of the scene is projected onto a plurality of pixels of a display from a first point-of-view. The pixels are partitioned into a plurality of tiles. A foveal region of highest resolution is defined for the image as displayed, wherein a first subset of pixels is assigned to the foveal region, and wherein a second subset of pixels is assigned to a peripheral region that is outside of the foveal region. A first set of light sources is determined from the plurality of light sources that affect one or more objects displayed in a first tile that is in the peripheral region. At least two light sources from the first set is clustered into a first aggregated light source affecting the first tile when rendering the image in pixels of the first tile.

First claim

What is claimed is:

1. A method for implementing a graphics pipeline, comprising:
determining a plurality of light sources affecting a virtual scene, wherein each of the plurality of light sources is a virtual light source generated during execution of an application to render a plurality of images of the virtual scene in the graphics pipeline;
projecting geometries of objects of an image of the virtual scene onto a plurality of pixels of a display from a first point-of-view;
partitioning a plurality of pixels of the display into a plurality of tiles;
defining a foveal region of highest resolution for the image as displayed, wherein a first subset of pixels is assigned to the foveal region, and wherein a second subset of pixels is assigned to a peripheral region that is outside of the foveal region;
determining a first set of light sources from the plurality of light sources that affect one or more objects displayed in a first tile that is in the peripheral region; and
clustering at least two light sources from the first set into a first aggregated light source affecting the first tile when rendering the image in pixels of the first tile.

2. The method of claim 1 , further comprising:
generating a plurality of clusters of light sources from the plurality of light sources using a nearest neighbor search or a k-means clustering technique, wherein each cluster of light sources is associated with a corresponding aggregated light source, wherein each cluster of light sources comprises at least one light source.

3. The method of claim 2 , further comprising:
constraining the number of aggregated light sources affecting the first tile to a target number;
constraining the maximum distance between two light sources in a cluster to a maximum distance; and
minimizing a plurality of cluster errors for the plurality of clusters of light sources; and
for each cluster of light sources, generating a corresponding aggregated light source.

4. The method of claim 3 , wherein the generating a corresponding aggregated light source further comprises:
determining an aggregated location of the corresponding aggregated light source by averaging locations of light sources in the corresponding cluster of light sources;
determining an aggregated intensity of the corresponding aggregated light source by summing the intensities of light sources in the corresponding cluster of light sources; and
determining an aggregated color of the corresponding aggregated light source by performing a weighted sum of colors for light sources in the corresponding cluster of light sources.

5. The method of claim 3 , wherein the cluster error corresponding to each cluster in the plurality of clusters of light sources comprises an iterative sum of the distances between a current light source and each of the other light sources, wherein the current light source comprises a different light source in the cluster for each iteration.

6. The method of claim 1 , further comprising:
constraining the number of aggregated light sources affecting the first tile to a target number;
determining that the first set of light sources is less than the target number; and
expanding a size of the first tile to meet the target number.

7. The method of claim 1 , wherein the display comprises a head mounted display (HMD).

8. The method of claim 1 , wherein the foveal region in the image corresponds to a static area that is centered in the display.

9. The method of claim 1 , wherein the foveal region in the image corresponds to where on the display an eye of the user is directed.

10. A computer system comprising:
a processor; and
memory coupled to the processor and having stored therein instructions that, if executed by the computer system, cause the computer system to execute a method for implementing a graphics pipeline, comprising:
determining a plurality of light sources affecting a virtual scene, wherein each of the plurality of light sources is a virtual light source generated during execution of an application to render a plurality of images of the virtual scene in the graphics pipeline;
projecting geometries of objects of an image of the virtual scene onto a plurality of pixels of a display from a first point-of-view;
partitioning a plurality of pixels of the display into a plurality of tiles;
defining a foveal region of highest resolution for the image as displayed, wherein a first subset of pixels is assigned to the foveal region, and wherein a second subset of pixels is assigned to a peripheral region that is outside of the foveal region;
determining a first set of light sources from the plurality of light sources that affect one or more objects displayed in a first tile that is in the peripheral region; and
clustering at least two light sources from the first set into a first aggregated light source affecting the first tile when rendering the image in pixels of the first tile.

11. The computer system of claim 10 , wherein the method further comprises:
generating a plurality of clusters of light sources from the plurality of light sources using a nearest neighbor search or a k-means clustering technique, wherein each cluster of light sources is associated with a corresponding aggregated light source, wherein each cluster of light sources comprises at least one light source.

12. The computer system of claim 11 , wherein the method further comprises:
constraining the number of aggregated light sources affecting the first tile to a target number;
constraining the maximum distance between two light sources in a cluster to a maximum distance; and
minimizing a plurality of cluster errors for the plurality of clusters of light sources; and
for each cluster of light sources, generating a corresponding aggregated light source.

13. The computer system of claim 12 , wherein the generating a corresponding aggregated light source further comprises:
determining an aggregated location of the corresponding aggregated light source by averaging locations of light sources in the corresponding cluster of light sources;
determining an aggregated intensity of the corresponding aggregated light source by summing the intensities of light sources in the corresponding cluster of light sources; and
determining an aggregated color of the corresponding aggregated light source by performing a weighted sum of colors for light sources in the corresponding cluster of light sources.

14. The computer system of claim 12 , wherein the cluster error corresponding to each cluster in the plurality of clusters of light sources comprises an iterative sum of the distances between a current light source and each of the other light sources, wherein the current light source comprises a different light source in the cluster for each iteration.

15. The computer system of claim 10 , wherein the method further comprises:
constraining the number of aggregated light sources affecting the first tile to a target number;
determining that the first set of light sources is less than the target number; and
expanding a size of the first tile to meet the target number.

16. The computer system of claim 10 , wherein the display comprises a head mounted display (HMD).

17. A non-transitory computer-readable medium storing a computer program for implementing a graphics pipeline, the computer-readable medium comprising:
program instructions for determining a plurality of light sources affecting a virtual scene, wherein each of the plurality of light sources is a virtual light source generated during execution of an application to render a plurality of

Assignees

• Sony Interactive Entertainment Inc

Inventors

• Young Andrew
• Ho Chris
• Stafford Jeffrey Roger

Classifications

• G06T15/50

  Lighting effects · CPC title

• G06T2210/36

  Level of detail · CPC title

• G06T11/10 Primary

  Texturing; Colouring; Generation of textures or colours (retouching, inpainting or scratch removal G06T5/77) · CPC title

• G06T13/80 Primary

  Two-dimensional [2D] animation, e.g. using sprites · CPC title

• G06T1/20

  Processor architectures; Processor configuration, e.g. pipelining · CPC title