Virtual photogrammetry

Patent metadata

Field	Value
Publication number	US-11625894-B2
Application number	US-202117204169-A
Country	US
Kind code	B2
Filing date	Mar 17, 2021
Priority date	Jul 13, 2018
Publication date	Apr 11, 2023
Grant date	Apr 11, 2023

Abstract

Multiple snapshots of a scene are captured within an executing application (e.g., a video game). When each snapshot is captured, associated color values per pixel and a distance or depth value z per pixel are stored. The depth information from the snapshots is accessed, and a point cloud representing the depth information is constructed. A mesh structure is constructed from the point cloud. The light field(s) on the surface(s) of the mesh structure are calculated. A surface light field is represented as a texture. A renderer uses the surface light field with geometry information to reproduce the scene captured in the snapshots. The reproduced scene can be manipulated and viewed from different perspectives.

First claim

What is claimed is:

1. A method comprising:
receiving, using one or more processing units, data representative of one or more depth values corresponding to a one or more images of at least one object captured from one or more perspectives within a virtual scene;
generating, using the one or more processing units, a mesh structure corresponding to the at least one object based at least in part on the one or more depth values;
determining, using the one or more processing units, one or more lighting characteristics associated with the mesh structure, the one or more lighting characteristics being encoded using spherical harmonics; and
rendering, using the one or more processing units, a three-dimensional representation of the at least one object based at least in part on the mesh structure and the one or more lighting characteristics.

2. The method of claim 1 , wherein the one or more images are captured using one or more virtual cameras within the virtual scene.

3. The method of claim 1 , further comprising:
generating a point cloud based at least in part on the one or more depth values,
wherein the generating the mesh structure is executed based at least in part on the point cloud.

4. The method of claim 1 , wherein the data is further representative of one or more color values corresponding to one or more pixels of the one or more images and the one or more depth values correspond to one or more pixels of the one or more images.

5. The method of claim 1 , wherein using the spherical harmonics comprises encoding at least one of incoming light, emitted light, or reflected light.

6. The method of claim 1 , further comprising:
generating a texture representation based at least in part on the one or more lighting characteristics,
wherein the rendering the three-dimensional representation is based at least in part on the texture representation.

7. The method of claim 1 , wherein the generating the mesh structure comprises:
for a first point in a point cloud generated based at least in part on the one or more depth values:
generating a polygon that is centered around the first point and is perpendicular to a normal for the first point;
cutting the polygon against a neighbor point of the first point in a middle of a segment connecting the first point and the neighbor point to generate a cut polygon; and
triangulating the cut polygon.

8. The method of claim 1 , wherein the generating the mesh structure includes executing a hole filler algorithm to fill one or more holes of the mesh structure.

9. The method of claim 8 , wherein the hole filler algorithm includes:
identifying a border edge on a border of a hole in the mesh structure by at least determining there is not an adjacent edge of a neighboring triangle to at least one edge of at least one triangle in the mesh structure;
forming at least one loop comprising the border edge, wherein the at least one loop corresponds to the hole in the mesh structure; and
triangulating of the at least one loop.

10. A processor comprising:
one or more circuits to render a three-dimensional representation of at least one object in a virtual scene from one or more images captured from one or more perspectives within the virtual scene, wherein rendering the three-dimensional representation is executed based at least in part on: (i) a mesh structure generated based at least in part on one or more depth values associated with the one or more images; and (2) one or more lighting characteristics associated with the mesh structure, wherein a first lighting characteristic of the one or more lighting characteristics is determined based at least in part on spherical harmonics.

11. The processor of claim 10 , wherein the mesh structure is generated based at least in part on a point cloud generated using the depth values.

12. The processor of claim 10 , wherein the one or more depth values are associated with one or more pixels of the one or more images.

13. The processor of claim 10 , wherein the processor includes one or more parallel processing units, and the mesh structure is generated using parallel processing of one or more blocks associated with the mesh structure.

14. The processor of claim 10 , wherein a second lighting characteristic of the one or more lighting characteristics is represented using a texture map.

15. A system comprising:
one or more processing units;
one or more memory devices storing instructions that, when executed using the one or more processing units, cause the one or more processing units to execute instructions comprising:
receiving data representative of one or more images of a virtual scene;
generating a point cloud based at least in part on the data;
generating a mesh structure based at least in part on the point cloud;
generating a texture representation of first one or more lighting characteristics associated with the mesh structure by at least delighting an object within the virtual scene; and
rendering a three-dimensional representation based at least in part on the mesh structure and the texture representation.

16. The system of claim 15 , wherein the one or more images are captured using a plurality of virtual cameras having different fields of view within the virtual scene.

17. The system of claim 15 , wherein the operations further comprise determining second one or more lighting characteristics associated with the mesh structure using spherical harmonics, wherein the rendering the three-dimensional representation is further based on the spherical harmonics.

18. The system of claim 15 , wherein the generating the mesh structure comprises:
for a point in the point cloud:
generating a polygon that is centered around the point and is perpendicular to a normal for the point;
cutting the polygon against a neighbor point of the point in a middle of a segment connecting the point and the neighbor point to generate a cut polygon; and
triangulating the cut polygon.

19. The system of claim 15 , wherein the generating the mesh structure includes:
determining a border edge on a border of a hole in the mesh structure by at least identifying at least one edge of at least one triangle in the mesh structure that does not include an adjacent edge of a neighboring triangle;
forming at least one loop comprising the border edge; and
triangulating the at least one loop.

20. The system of claim 15 , wherein the one or more processing units include one or more parallel processing units, and wherein the generating the mesh structure is executed, at least in part, in parallel using the one or more parallel processing units.

Assignees

• Nvidia Corp

Inventors

• Duka Dmitry
• Lebaredian Rev
• Small Jonathan
• Shutov Ivan

Classifications

• G06T15/20

  Perspective computation · CPC title

• G06T2210/56

  Particle system, point based geometry or rendering · CPC title

• G06T17/00 Primary

  Three-dimensional [3D] modelling for computer graphics · CPC title

• G06T7/596

  from three or more stereo images · CPC title

• G06T15/00 Primary

  Three-dimensional [3D] image rendering · CPC title