Geometry-aware augmented reality effects with real-time depth map

Patent metadata

Field	Value
Publication number	US-12236541-B2
Application number	US-202017998736-A
Country	US
Kind code	B2
Filing date	May 29, 2020
Priority date	May 29, 2020
Publication date	Feb 25, 2025
Grant date	Feb 25, 2025

Abstract

Techniques of introducing virtual objects into a physical environment of AR system include displacing vertices of a mesh representing the physical environment based on a live depth map. For example, an AR system generates a mesh template, i.e., an initial mesh with vertices that represents a physical environment and a depth map that indicates a geometry of real objects within the physical environment. The AR system is configured to represent the real objects in the physical environment by displacing the vertices of the mesh based on depth values of the depth map and parameter values of a pinhole camera model. The depth values may be taken from the perspective of an illumination source in the physical environment.

First claim

What is claimed is:

1. A method, comprising:
generating a triangular mesh representing a physical environment and a depth map of the physical environment, the triangular mesh including a plurality of vertices, the depth map including a plurality of depth values;
performing a displacement operation on the plurality of vertices of the triangular mesh including removing a vertex of the plurality of vertices of the triangular mesh in response to the depth map satisfying a criterion to produce a plurality of displaced vertices representing a geometry of at least one real object within the physical environment;
receiving virtual object data representing a virtual object configured to be displayed with the at least one real object in the physical environment; and
displaying the virtual object in the physical environment on a display to produce a displayed virtual object, the displayed virtual object having a difference from the virtual object according to the plurality of displaced vertices.

2. The method as in claim 1 , wherein performing the displacement operation includes:
displacing each of the plurality of vertices of the triangular mesh according to a pinhole camera model.

3. The method as in claim 1 , wherein the criterion includes a difference being greater than a threshold, the difference being between the depth value of the depth map corresponding to the vertex and an average of the depth value of the depth map corresponding to the vertex and the depth values corresponding to a set of neighboring vertices of the plurality of vertices.

4. The method as in claim 1 , wherein the triangular mesh further includes a plurality of indices indicating a connectivity of the triangular mesh, each of the plurality of indices corresponding to a respective vertex of the plurality of vertices of the triangular mesh, the connectivity of the triangular mesh being constant over time.

5. The method as in claim 4 , wherein the plurality of indices are arranged in a plurality of triplets of indices, each of the plurality of triplets of indices representing a triangle of the triangular mesh and being arranged in an order representing a specified winding order.

6. The method as in claim 4 , wherein generating the triangular mesh includes:
storing the plurality of vertices and the plurality of indices of the triangular mesh in a first buffer in a memory in a central processing unit (CPU); and
copying the plurality of vertices and the plurality of indices to a second buffer in a memory of a graphics processing unit (GPU).

7. The method as in claim 1 , wherein generating the depth map includes:
generating the depth values of the depth map along a ray emanating from a light source in a vicinity of the physical environment.

8. The method as in claim 1 , further comprising, prior to receiving the virtual object data:
rendering the triangular mesh having the plurality of displaced vertices as a transparent mesh to the display; and
rendering first shadows on the transparent mesh based on the depth map.

9. The method as in claim 8 , wherein displaying the virtual object in the physical environment on the display includes:
rendering second shadows on the triangular mesh based on the displayed virtual object.

10. The method as in claim 1 , further comprising:
generating a mesh collider based on the triangular mesh, the mesh collider being configured to detect collisions between the at least one real object and the virtual object, the mesh collider including a set of vertices.

11. The method as in claim 10 , further comprising:
determining a field of view (FOV) of a camera within which the virtual object in the physical environment is displayed;
extending a boundary of the mesh collider in response to the virtual object moving out of the FOV of the camera.

12. The method as in claim 10 , wherein generating the mesh collider includes:
computing, as a normal to the mesh collider in a vicinity of a vertex of the set of vertices, a cross product of orthogonal vectors formed by neighboring vertices of the set of vertices.

13. A computer program product comprising a nontransitive storage medium, the computer program product including code that, when executed by processing circuitry of a server computing device, causes the processing circuitry to perform a method, the method comprising:
generating a triangular mesh representing a physical environment and a depth map of the physical environment, the triangular mesh including a plurality of vertices, the depth map including a plurality of depth values;
performing a displacement operation on the plurality of vertices of the triangular mesh including removing a vertex of the plurality of vertices of the triangular mesh in response to the depth map satisfying a criterion to produce a plurality of displaced vertices representing a geometry of at least one real object within the physical environment;
receiving virtual object data representing a virtual object configured to be displayed with the at least one real object in the physical environment; and
displaying the virtual object in the physical environment on a display to produce a displayed virtual object, the displayed virtual object having a difference from the virtual object according to the plurality of displaced vertices.

14. The computer program product as in claim 13 , wherein the criterion includes a difference being greater than a threshold, the difference being between the depth value of the depth map corresponding to the vertex and an average of the depth value of the depth map corresponding to the vertex and the depth values corresponding to a set of neighboring vertices of the plurality of vertices.

15. The computer program product as in claim 13 , wherein the triangular mesh further includes a plurality of indices indicating a connectivity of the triangular mesh, each of the plurality of indices corresponding to a respective vertex of the plurality of vertices of the triangular mesh, the connectivity of the triangular mesh being constant over time.

16. The computer program product as in claim 13 , wherein generating the depth map includes:
generating the depth values of the depth map along a ray emanating from a light source in a vicinity of the physical environment.

17. The computer program product as in claim 13 , further comprising:
generating a mesh collider based on the triangular mesh, the mesh collider being configured to detect collisions between the at least one real object and the virtual object, the mesh collider including a set of vertices.

18. An electronic apparatus, the electronic apparatus comprising:
memory; and
controlling circuitry coupled to the memory, the controlling circuitry being configured to:
generate a triangular mesh representing a physical environment and a depth map of the physical environment, the triangular mesh including a plurality of vertices, the depth map including a plurality of depth values;
perform a displacement operation on the plurality of vertices of the triangular mesh including removing a vertex of the plurality of vertices of the triangular mesh in response to the depth map satisfying a criterion to produce a plurality of displaced vertices representing a geometry of at least one real object within the physical environment;
receive virtual object data representing a virtual object configured to be displayed with the at least one real object in the physical environment; and
display the virtual object in the physical environment on a display to produce a displayed virtual object, the displayed virtual object

Assignees

• Google Llc

Inventors

• Kim David
• Du Ruofei

Classifications

• G06T2215/16

  Using real world measurements to influence rendering · CPC title

• G06T2215/12

  Shadow map, environment map · CPC title

• G06T2210/21

  Collision detection, intersection · CPC title

• G06T2207/10028

  Range image; Depth image; 3D point clouds · CPC title

• G06T2200/28

  involving image processing hardware · CPC title