Virtual photogrammetry

Patent metadata

Field	Value
Publication number	US-10984587-B2
Application number	US-201916434972-A
Country	US
Kind code	B2
Filing date	Jun 7, 2019
Priority date	Jul 13, 2018
Publication date	Apr 20, 2021
Grant date	Apr 20, 2021

Abstract

Multiple snapshots of a scene are captured within an executing application (e.g., a video game). When each snapshot is captured, associated color values per pixel and a distance or depth value z per pixel are stored. The depth information from the snapshots is accessed, and a point cloud representing the depth information is constructed. A mesh structure is constructed from the point cloud. The light field(s) on the surface(s) of the mesh structure are calculated. A surface light field is represented as a texture. A renderer uses the surface light field with geometry information to reproduce the scene captured in the snapshots. The reproduced scene can be manipulated and viewed from different perspectives.

First claim

What is claimed is:

1. A method comprising:
accessing information stored in memory of a computer system, the information comprising snapshots of one or more objects in a scene generated with a software application, wherein the snapshots are captured from different perspectives relative to the scene;
generating a point cloud based on the snapshots;
generating a mesh structure based on the point cloud;
determining a light field on a surface of the mesh structure; and
storing the mesh structure and the light field in the memory of the computer system.

2. The method of claim 1 , wherein the information further comprises color values per pixel of the snapshots and a depth value per pixel of the snapshots.

3. The method of claim 1 , wherein the determining the light field comprises using spherical harmonic transforms.

4. The method of claim 1 , further comprising reproducing the scene in virtual three dimensions using the light field and the mesh structure to generate a reproduced scene.

5. The method of claim 4 , further comprising displaying the reproduced scene from different perspectives based at least in part on data representative of user inputs.

6. The method of claim 1 , wherein the software application comprises a video game.

7. The method of claim 1 , wherein the generating the mesh structure comprises:
for each point in the point cloud, generating a polygon that is centered around a respective point and is perpendicular to a normal for the respective point;
cutting the polygon against a neighbor point by placing a cutting plane in a middle of a segment connecting the respective point and the neighbor point; and
triangulating each polygon resulting from said cutting to generate the mesh structure.

8. The method of claim 7 , further comprising:
checking each edge of triangles in the mesh structure to determine whether there is an adjacent edge of a neighboring triangle, wherein when there is not the adjacent edge the edge is identified as being a border edge on a border of a hole in the mesh structure;
forming loops comprising edges identified as being border edges based on the checking, wherein each of the loops corresponds to a respective hole in the mesh structure; and
triangulating each of the loops.

9. A computer system, comprising:
one or more processors;
a display coupled to the one or more processors; and
memory coupled to the one or more processors, the memory having stored thereon instructions that, when executed by the computer system, cause the computer system to execute operations comprising:
accessing information stored in the memory, the information comprising snapshots of one or more objects in a scene generated with a software application, wherein the snapshots are captured from different perspectives relative to the scene;
generating a point cloud based on the snapshots;
generating a mesh structure based on the point cloud;
determining a light field on a surface of the mesh structure; and
storing the mesh structure and the light field.

10. The computer system of claim 9 , wherein the information further comprises color values per pixel of the snapshots and a depth value per pixel of the snapshots.

11. The computer system of claim 9 , wherein the light field is generated using spherical harmonic transforms.

12. The computer system of claim 9 , wherein the operations further comprise rendering and displaying the scene in virtual three dimensions using the light field and the mesh structure.

13. The computer system of claim 9 , wherein the operations further comprise:
for each point in the point cloud, generating a polygon that is centered around a respective point and is perpendicular to a normal for the respective point;
cutting the polygon against a neighbor point by placing a cutting plane in a middle of a segment connecting the respective point and the neighbor point; and
triangulating each polygon resulting from said cutting to generate the mesh structure.

14. The computer system of claim 13 , wherein the operations further comprise:
checking each edge of triangles in the mesh structure to determine whether there is an adjacent edge of a neighboring triangle, wherein when there is not the adjacent edge the edge is identified as being a border edge on a border of a hole in the mesh structure;
forming loops comprising edges identified as being border edges based on the checking, wherein each of the loops corresponds to a respective hole in the mesh structure; and
triangulating each of the loops.

15. A non-transitory computer-readable storage medium having computer-executable instructions that, when executed, perform a method comprising:
accessing information stored in memory of a computer system, the information comprising snapshots of one or more objects in a scene generated with a software application, wherein the snapshots are captured from different perspectives relative to the scene;
generating a point cloud based on the snapshots;
generating a mesh structure based on the point cloud;
determining a light field on a surface of the mesh structure; and
storing the mesh structure and the light field in the memory of the computer system.

16. The non-transitory computer-readable storage medium of claim 15 , wherein the information further comprises color values per pixel of the snapshots and a depth value per pixel of the snapshots.

17. The non-transitory computer-readable storage medium of claim 15 , wherein the determining the light field comprises using spherical harmonic transforms.

18. The non-transitory computer-readable storage medium of claim 15 , wherein the method further comprises rendering and displaying the scene in virtual three dimensions using the light field and the mesh structure.

19. The non-transitory computer-readable storage medium of claim 15 , wherein the method further comprises:
for each point in the point cloud, generating a polygon that is centered around a respective point and is perpendicular to a normal for the respective point;
cutting the polygon against a neighbor point by placing a cutting plane in a middle of a segment connecting the respective point and the neighbor point; and
triangulating each polygon resulting from said cutting to generate the mesh structure.

20. The non-transitory computer-readable storage medium of claim 19 , wherein the method further comprises:
checking each edge of triangles in the mesh structure to determine whether there is an adjacent edge of a neighboring triangle, wherein when there is not the adjacent edge the edge is identified as being a border edge on a border of a hole in the mesh structure;
forming loops comprising edges identified as being border edges based on the checking, wherein each of the loops corresponds to a respective hole in the mesh structure; and
triangulating each of the loops.

Assignees

• Nvidia Corp

Inventors

• Duka Dmitry
• Lebaredian Rev
• Small Jonathan
• Shutov Ivan

Classifications

• A63F13/52

  involving aspects of the displayed game scene · CPC title

• G06T15/50

  Lighting effects · CPC title

• G06T2210/56

  Particle system, point based geometry or rendering · CPC title

• G06T15/20

  Perspective computation · CPC title

• G06T17/20 Primary

  Finite element generation, e.g. wire-frame surface description, {tesselation} · CPC title