Real-time hand modeling and tracking using convolution models

Patent metadata

Field	Value
Publication number	US-11568601-B2
Application number	US-201716325372-A
Country	US
Kind code	B2
Filing date	Aug 14, 2017
Priority date	Aug 14, 2016
Publication date	Jan 31, 2023
Grant date	Jan 31, 2023

Abstract

Technologies are provided herein for modeling and tracking physical objects, such as human hands, within a field of view of a depth sensor. A sphere-mesh model of the physical object can be created and used to track the physical object in real-time. The sphere-mesh model comprises an explicit skeletal mesh and an implicit convolution surface generated based on the skeletal mesh. The skeletal mesh parameterizes the convolution surface and distances between points in data frames received from the depth sensor and the sphere-mesh model can be efficiently determined using the skeletal mesh. The sphere-mesh model can be automatically calibrated by dynamically adjusting positions and associated radii of vertices in the skeletal mesh to fit the convolution surface to a particular physical object.

First claim

We claim:

1. A method for modeling and tracking a physical object, the method comprising:
generating a sphere-mesh model of the physical object, wherein the sphere-mesh model comprises an explicit skeletal mesh comprising multiple connected vertices and an implicit surface generated based on the skeletal mesh, wherein generating the implicit surface based on the skeletal mesh comprises creating sphere-mesh primitives comprising spheres with radii that are properties of the multiple vertices of the explicit skeletal mesh, and centering the spheres on the multiple vertices;
receiving a data frame from a sensor camera associated with a camera direction, wherein the data frame comprises depth information for points within a field of view of the sensor camera from the camera direction, wherein the physical object is within the field of view of the sensor camera; and
determining a distance between one of the points within the field of view of the sensor camera and the implicit surface of the sphere-mesh model by determining a distance between the one of the points and the explicit skeletal mesh of the sphere-mesh model, wherein the determining the distance includes finding a closest point on the sphere-mesh model and replacing a back facing closest point with a front facing closest point.

2. The method of claim 1 , wherein the determining the distance between the one of the points within the field of view and the implicit surface comprises determining a distance between the one of the points and the sphere-mesh primitives.

3. The method of claim 1 , wherein the determining the distance between the one of the points and the sphere-mesh primitives comprises using the vertices and the associated radii of the sphere-mesh primitives to determine a point of the implicit surface closest to the one of the points.

4. The method of claim 1 , wherein the generating the implicit surface further comprises creating a pill primitive as a convex hull of two of the sphere-mesh primitives for two of the connected vertices.

5. The method of claim 1 , wherein the generating the implicit surface further comprises creating a wedge primitive as a convex hull of three of the sphere-mesh primitives for three of the connected vertices.

6. The method of claim 1 , further comprising:
generating a sphere-mesh model template, wherein the generating the sphere-mesh model template comprises adjusting a topological structure of the sphere-mesh model to match a topological structure of the physical object, wherein the adjusting comprises adjusting a position of at least one of the multiple vertices and/or adjusting at least one of the radii associated with the multiple vertices.

7. The method of claim 1 , further comprising calibrating the sphere-mesh model, wherein the calibrating comprises:
adjusting positions of the multiple vertices and adjusting the associated radii to match the implicit surface to a geometric representation of the physical object.

8. The method of claim 7 , further comprising:
obtaining the geometric representation of the physical object from the sensor camera.

9. The method of claim 1 , wherein the physical object comprises a human body part.

10. The method of claim 1 , wherein the physical object comprises a human hand.

11. The method of claim 10 , further comprising determining at least one convolution surface corresponding to the human hand and adjusting at least one radius and one vertex of the sphere-mesh model to alter the at least one convolution surface.

12. A system, comprising:
a depth sensor configured to generate depth points within a field of view of the depth sensor; and
a computing device connected to the depth sensor and configured to:
generate a sphere-mesh model of a physical object, wherein the sphere-mesh model comprises an explicit skeletal mesh comprising multiple connected vertices and an implicit surface generated based on the skeletal mesh, wherein generating the implicit surface based on the skeletal mesh comprises creating sphere-mesh primitives comprising spheres with radii that are properties of the multiple vertices of the explicit skeletal mesh, and centering the spheres on the multiple vertices;
receive the depth points from the depth sensor; and
determine a distance between one of the depth points and the implicit surface of the sphere-mesh model by determining a distance between the one of the depth points and the explicit skeletal mesh of the sphere-mesh model, wherein the determining the distance includes finding a closest point on the sphere-mesh model and replacing a back facing closest point with a front facing closest point.

13. The system of claim 12 , wherein the determining the distance between the one of the depth points and the implicit surface comprises determining a distance between the one of the depth points and the sphere-mesh primitives.

14. The system of claim 12 , wherein the determining the distance between the one of the depth points and the sphere-mesh primitives comprises using the vertices and the associated radii of the sphere-mesh primitives to determine a point of the implicit surface closest to the one of the depth points.

15. The system of claim 12 , wherein the generating the implicit surface further comprises creating a pill primitive as a convex hull of two of the sphere-mesh primitives for two of the connected vertices.

16. The system of claim 12 , wherein the generating the implicit surface further comprises creating a wedge primitive as a convex hull of three of the sphere-mesh primitives for three of the connected vertices.

17. The system of claim 12 , wherein the computing device is further configured to:
generate a sphere-mesh model template, wherein the generating the sphere-mesh model template comprises adjusting a topological structure of the sphere-mesh model to match a topological structure of the physical object, wherein the adjusting comprises adjusting a position of at least one of the multiple vertices and/or adjusting at least one of the radii associated with the multiple vertices.

18. The system of claim 12 , wherein the computing device is further configured to calibrate the sphere-mesh model, wherein the calibrating comprises:
adjusting positions of the multiple vertices and adjusting the associated radii to match the implicit surface to a geometric representation of the physical object.

19. The system of claim 18 , wherein the computing device is further configured to obtain the geometric representation of the physical object from the depth sensor.

20. The system of claim 12 , wherein the physical object comprises a human body part.

21. The system of claim 12 , wherein the physical object comprises a human hand.

22. The system of claim 12 , wherein the closest point on the sphere-mesh model is determined to be front-facing based on a surface normal to a convolution surface associated with the sphere-mesh model.

23. One or more computer-readable media storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations, the operations comprising:
generating a sphere-mesh model of a physical object, wherein the sphere-mesh model comprises an explicit skeletal mesh comprising multiple connected vertices and an implicit surface generated based on the skeletal mesh, wherein generating the implicit surface based on the skeletal mesh comprises creating sphere-mesh primitives comprising spheres with radii that are properties of the mult

Assignees

• Uvic Ind Partnerships Inc

Inventors

• Tagliasacchi Andrea
• Tkach Anastasia
• Pauly Mark

Classifications

• G06T7/251

  involving models · CPC title

• G06T2207/10028

  Range image; Depth image; 3D point clouds · CPC title

• G06V40/28

  Recognition of hand or arm movements, e.g. recognition of deaf sign language (static hand signs G06V40/113) · CPC title

• G06T2207/30196

  Human being; Person · CPC title

• G06T2207/10016

  Video; Image sequence · CPC title