Method and system for determining 3D object poses and landmark points using surface patches

Patent metadata

Field	Value
Publication number	US-10515259-B2
Application number	US-201514632466-A
Country	US
Kind code	B2
Filing date	Feb 26, 2015
Priority date	Feb 26, 2015
Publication date	Dec 24, 2019
Grant date	Dec 24, 2019

Abstract

A method and system determine a three-dimensional (3D) pose of an object and 3D locations of landmark points of the object by first obtaining a 3D point cloud of the object. 3D surface patches are extracted from the 3D point cloud, and a parametric model is fitted to each 3D surface patch to determine a set of descriptors. A set of correspondences between the set of descriptors and a set of descriptors of patches extracted from 3D point clouds of objects from the same object class with known 3D poses and known 3D locations of landmark points is determined. Then, the 3D pose of the object and 3D locations of the landmark points of the object are estimated from the set of correspondences.

First claim

We claim:

1. A method for determining a three-dimensional (3D) pose of an object and 3D locations of landmark points of the object, comprising steps of:
obtaining a 3D point cloud of the object;
sampling 3D surface patches from the 3D point cloud, each 3D surface patch includes a 3D center point, a 3D normal vector and 3D geometric surface, and each 3D surface patch is presented as a descriptor including parameters of a parametric model that fit to each 3D surface patch of the 3D point cloud, to obtain a set of descriptors;
determining a set of correspondences between the set of descriptors and a set of training descriptors of patches extracted from training objects from the same object class as the obtained object with known 3D poses and known 3D locations of landmark points;
determining for each correspondence from the set of correspondences a 3D rotation and a 3D translation, that map the 3D normal vector of the training 3D surface patch to the 3D normal vector of the sampled 3D surface patch, and using the 3D rotation and 3D translation to transform the training vectors pointing to the 3D locations of the landmark points on the training object to the landmark locations on the obtained 3D point cloud, to obtain a set of 3D rotations, 3D translations and 3D vectors pointing to the landmark points of the object; and
estimating the 3D pose of the object, that includes a 3D rotation and a 3D translation, and 3D locations of the landmark points of the object from the estimates contained in the set of 3D rotations, 3D translations and 3D vectors pointing to the landmark points of the object, wherein the steps are performed in a processor connected to a memory storing the 3D point cloud.

2. The method of claim 1 , wherein the object is a human head and the landmark points are facial landmarks.

3. The method of claim 1 , wherein the 3D point cloud is obtained by computing a 3D point cloud from a depth image.

4. The method of claim 3 , wherein the depth image is acquired by a depth sensor.

5. The method of claim 1 , wherein the determining is performed using an approximate nearest neighbor algorithm.

6. The method of claim 1 , wherein the 3D point cloud is generated synthetically.

7. The method of claim 1 , wherein the 3D surface patches are triangular, and the parametric model is a sub-triangle parametric model.

8. The method of claim 1 , wherein the 3D surface patches are selected from random locations and random orientations on a surface defined by the 3D point cloud.

9. The method of claim 7 , further comprising:
filling in holes in the parametric model using neighboring sub-triangles.

10. The method of claim 1 , wherein the estimating is performed using a voting technique.

11. The method of claim 10 , wherein inconsistent votes are filtered by performing joint clustering in rotation and centroid voting spaces.

12. The method of claim 10 , wherein the results of the estimating are used by an augmented reality system for cars that uses a head-up display.

13. A system for determining a three-dimensional (3D) pose of an object and 3D locations of landmarks of the object, comprising:
a memory having stored therein information about historical 3D training objects, including training descriptors representing 3D surface patches of training objects, 3D pose and 3D landmark point locations of the training objects; and
a processor, connected to the memory, is configured to:
access a 3D point cloud of the object;
extract 3D surface patches from the 3D point cloud, each 3D surface patch includes a 3D center point, a 3D normal vector and 3D geometric surface, and each 3D surface patch is presented as a descriptor including parameters of a parametric model that fit to each 3D surface patch of the 3D point cloud, to obtain a set of descriptors, wherein the extracted 3D surface patches include a planar triangular shape and 3D points located above, below, or both, of a surface of the planar triangular shape;
determine a set of correspondences between the set of descriptors and a set of training descriptors of patches extracted from training objects from the same object class as the accessed object with known 3D poses and known 3D locations of landmarks;
determine for each correspondence from the set of correspondences a 3D rotation and a 3D translation, that map the 3D normal vector of the training 3D surface patch to the 3D normal vector of the sampled 3D surface patch, and using the 3D rotation and 3D translation to transform the training vectors pointing to the 3D locations of the landmark points on the training object to the landmark locations on the obtained 3D point cloud, to obtain a set of 3D rotations, 3D translations and 3D vectors pointing to the landmark points of the object; and
estimate the 3D pose of the object, that includes a 3D rotation and a 3D translation, and 3D locations of the landmarks of the object from the estimates contained in the set of 3D rotations, 3D translations and 3D vectors pointing to the landmark points of the object.

14. A data processing system for determining a three-dimensional (3D) pose of an object and 3D locations of landmark points of the object, comprising:
a data storage memory having stored therein information about historical 3D training objects, including training descriptors representing 3D surface patches of training objects, 3D pose and 3D landmark point locations of the training objects;
at least one processor, coupled to input interfaces, output interfaces and the data storage memory, the at least one processor is configured to:
access a 3D point cloud of the object;
extract 3D surface patches from the 3D point cloud, each 3D surface patch includes a 3D center point, a 3D normal vector and 3D geometric surface, and each 3D surface patch is presented as a descriptor including parameters of a sub-triangle parametric model that fit to each 3D surface patch of the 3D point cloud, to obtain a set of descriptors, the sub-triangle parametric model includes a base triangle and a set of sub-triangles wherein the extracted 3D surface patches include a planar triangular shape and 3D points located above, below, or both, of a surface of the planar triangular shape;
compare each descriptor to a set of training descriptors of the training descriptors stored in the data storage memory, so as to be matched to a training descriptor from the set of training descriptors;
determine a set of correspondences between the set of descriptors and a set of training descriptors of patches extracted from training objects from the same object class as the accessed object with known 3D poses and known 3D locations of landmarks;
determine for each correspondence from the set of correspondences a 3D rotation and a 3D translation, that map the 3D normal vector of the training 3D surface patch to the 3D normal vector of the sampled 3D surface patch, and using the 3D rotation and 3D translation to transform the training vectors pointing to the 3D locations of the landmark points on the training object to the landmark locations on the obtained 3D point cloud, to obtain a set of 3D rotations, 3D translations and 3D vectors pointing to the landmark points of the object; and
estimate the 3D pose of the object, that includes a 3D rotation and a 3D translation, and 3D locations of the landmark of the object from the estimates contained in the set of 3D rotations, 3D translations and 3D vectors pointing to the landmark points of the object.

Assignees

• Mitsubishi Electric Res Laboratories Inc

Inventors

• Jones Michael J
• Marks Tim
• Papazov Chavdar

Classifications

• G06T2207/30201

  Face · CPC title

• G06T7/75

  involving models · CPC title

• G06V10/82 Primary

  using neural networks · CPC title

• G06V10/764

  using classification, e.g. of video objects · CPC title

• G06V40/16

  Human faces, e.g. facial parts, sketches or expressions · CPC title