Display engine for post-rendering processing

Patent metadata

Field	Value
Publication number	US-11640691-B2
Application number	US-202217837540-A
Country	US
Kind code	B2
Filing date	Jun 10, 2022
Priority date	Nov 2, 2018
Publication date	May 2, 2023
Grant date	May 2, 2023

Abstract

In one embodiment, a computing system may access surfaces and texel data of an artificial reality scene. The surfaces may be generated based on a first viewing position of a viewer. The system may determine tiles on a display to test for a visibility of the surfaces from a second viewing position. The tiles may include first tiles that need more computational resources and second tiles that need less computational resources. The system may determine a tile order which interleaves the first and second tiles. The system may generate rays based on the tile order. The system may determine the visibility of the surfaces from the second viewing position based on the ray-surface intersections. The system may generate color values of a subframe based on the surface visibility and the texel data. The system may provide the color values to the display.

First claim

What is claimed is:

1. A method, by a computing system:
accessing one or more surfaces and texel data of an artificial reality scene, wherein the one or more surfaces are generated based on a first viewing position of a viewer;
determining a plurality of tiles on a display for which to test for a visibility of the one or more surfaces from a second viewing position of the viewer, wherein the plurality of tiles comprises one or more first tiles associated with one or more first areas of the display and one or more second tiles associated with one or more second areas of the display, and wherein the one or more first areas have higher level distortions than the one or more second areas;
determining an order for the plurality of tiles, wherein the order interleaves the one or more first tiles and the one or more second tiles;
generating rays based on the determined order of the plurality of tiles;
determining the visibility of the one or more surfaces of the artificial reality scene from the second viewing position based on intersections of the rays with the one or more surfaces;
generating color values of a subframe based on the visibility of the one or more surfaces and the texel data; and
providing the color values of the subframe to the display.

2. The method of claim 1 , wherein each of the one or more first tiles is predicted to have a longer processing time than a corresponding second tile of the one or more second tiles.

3. The method of claim 1 , wherein the one or more first areas are predicted to have a higher chromatic aberration than the one or more second areas.

4. The method of claim 1 , wherein the one or more first tiles are associated with one or more edge areas of the display, and wherein the one or more second tiles are associated with one or more central areas of the display.

5. The method of claim 1 , wherein determining the visibility of the one or more surfaces of the artificial reality scene from the second viewing position comprises:
applying a geometric transformation to the rays to modify their respective trajectories in the artificial reality scene; and
determining, based on the modified trajectories of the rays, intersections of the rays with the one or more surfaces in the artificial reality scene.

6. The method of claim 5 , wherein generating the color values of the subframe based on the visibility of the one or more surfaces and the texel data comprises:
mapping the intersections of the rays with the one or more surfaces to surface coordinates of a surface texture, wherein the color values of the subframe are determined by interpolating the texel data based on the mapped intersections.

7. The method of claim 1 , further comprising performing one or more graphical optimization processes on the color values, wherein the one or more graphical optimization processes comprise:
a brightness adjustment process;
a dithering process;
a foveated rendering process; or
a perception compression rendering process.

8. The method of claim 1 , wherein the computing system comprises a processing pipeline comprising a plurality of sub-processing units each associated with a separate memory unit, and wherein the plurality of sub-processing units receive data from a control unit through respective separate memory units.

9. The method of claim 1 , wherein the computing system comprises a processing unit comprising a buffer manager and a plurality of filter blocks each being associated with a color channel, wherein each filter block comprises a texel buffer and a pixel buffer, and wherein the color values are generated by parallelly sampling the texel data using respective texel buffers and pixel buffers.

10. The method of claim 1 , wherein the color values of the subframe are provided to the display using a display block comprising one or more row buffers, a pixel sequencer, and a brightness correction unit, wherein the one or more row buffers store pixel data, metadata, and foveation pattern data, wherein the pixel sequencer converts the pixel data from a tile order to a line order based on the foveation data, and wherein the brightness correction unit corrects brightness of the color values based on the pixel data and the metadata.

11. One or more computer-readable non-transitory storage media embodying software that is operable when executed to:
access one or more surfaces and texel data of an artificial reality scene, wherein the one or more surfaces are generated based on a first viewing position of a viewer;
determine a plurality of tiles on a display for which to test for a visibility of the one or more surfaces from a second viewing position of the viewer, wherein the plurality of tiles comprises one or more first tiles associated with one or more first areas of the display and one or more second tiles associated with one or more second areas of the display, and wherein the one or more first areas have higher level distortions than the one or more second areas;
determine an order for the plurality of tiles, wherein the order interleaves the one or more first tiles and the one or more second tiles;
generate rays based on the determined order of the plurality of tiles;
determine the visibility of the one or more surfaces of the artificial reality scene from the second viewing position based on intersections of the rays with the one or more surfaces;
generate color values of a subframe based on the visibility of the one or more surfaces and the texel data; and
provide the color values of the subframe to the display.

12. The media of claim 11 , wherein each of the one or more first tiles is predicted to have a longer processing time than a corresponding second tile of the one or more second tiles.

13. The media of claim 11 , wherein the one or more first areas are predicted to have a higher chromatic aberration than the one or more second areas.

14. The media of claim 11 , wherein the one or more first tiles are associated with one or more edge areas of the display, and wherein the one or more second tiles are associated with one or more central areas of the display.

15. A system comprising:
one or more processors; and
one or more computer-readable non-transitory storage media coupled to one or more of the processors and comprising instructions operable when executed by one or more of the processors to cause the system to:
access one or more surfaces and texel data of an artificial reality scene, wherein the one or more surfaces are generated based on a first viewing position of a viewer;
determine a plurality of tiles on a display for which to test for a visibility of the one or more surfaces from a second viewing position of the viewer, wherein the plurality of tiles comprises one or more first tiles associated with one or more first areas of the display and one or more second tiles associated with one or more second areas of the display, and wherein the one or more first areas have higher level distortions than the one or more second areas;
determine an order for the plurality of tiles, wherein the order interleaves the one or more first tiles and the one or more second tiles;
generate rays based on the determined order of the plurality of tiles;
determine the visibility of the one or more surfaces of the artificial reality scene from the second viewing position based on intersections of the rays with the one or more surfaces;
generate color values of a subframe based on the visibility of the one or more surfaces and the texel data; and
provide the color values of the subframe to the display.

16. The system of claim 15 , wherein each of the one or

Assignees

• Facebook Tech Llc
• Meta Platforms Tech Llc

Inventors

• Seiler Larry
• Nankervis Alexander
• Hunt Warren Andrew

Classifications

• G06T15/205 Primary

  Image-based rendering · CPC title

• G06F3/011

  Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00) · CPC title

• G06T15/02

  Non-photorealistic rendering · CPC title

• G06T15/06 Primary

  Ray-tracing · CPC title

• G06T15/40 Primary

  Hidden part removal · CPC title