Real-time, model-based object detection and pose estimation

Patent metadata

Field	Value
Publication number	US-2016379083-A1
Application number	US-201514749189-A
Country	US
Kind code	A1
Filing date	Jun 24, 2015
Priority date	Jun 24, 2015
Publication date	Dec 29, 2016
Grant date	—

Abstract

A system includes a memory and a processor configured to select a set of scene point pairs, to determine a respective feature vector for each scene point pair, to find, for each feature vector, a respective plurality of nearest neighbor point pairs in feature vector data of a number of models, to compute, for each nearest neighbor point pair, a respective aligning transformation from the respective scene point pair to the nearest neighbor point pair, thereby defining a respective model-transformation combination for each nearest neighbor point pair, each model-transformation combination specifying the respective aligning transformation and the respective model with which the nearest neighbor point pair is associated, to increment, with each binning of a respective one of the model-transformation combinations, a respective bin counter, and to select one of the model-transformation combinations in accordance with the bin counters to detect an object and estimate a pose of the object.

First claim

What is claimed is:

1 . A system comprising:
a memory in which feature vector instructions, matching instructions, and voting instructions are stored; and a processor coupled to the memory, the processor configured via execution of: the feature vector instructions to obtain a mesh for a scene input, to select a set of scene point pairs of the mesh, and to determine a respective feature vector for each scene point pair of the set of scene point pairs; the matching instructions to find, for each feature vector, a respective plurality of nearest neighbor point pairs in feature vector data of a plurality of models, the feature vector data of each model being indicative of a corresponding object of a plurality of objects, and further to compute, for each nearest neighbor point pair of the pluralities of nearest neighbor point pairs, a respective aligning transformation from the respective scene point pair to the nearest neighbor point pair, thereby defining a respective model-transformation combination for each nearest neighbor point pair, each model-transformation combination specifying the respective aligning transformation and the respective model with which the nearest neighbor point pair is associated; and the voting instructions to increment, with each binning of a respective one of the model-transformation combinations, a respective bin counter for the model-transformation combination, and further to select a number of the model-transformation combinations in accordance with the bin counters to detect a number of the objects in the scene input and estimate a pose of each detected object.

2 . The system of claim 1 , wherein the processor is further configured via the execution of the matching instructions to:
access a search tree database in which the feature vector data is stored; and query the search tree database to determine the plurality of nearest neighbor point pairs based on a respective distance between the feature vector of the respective scene point pair and the feature vector data of each model.

3 . The system of claim 1 , wherein the feature vector is descriptive of local geometry at each point of the scene point pair in a translationally and rotationally invariant manner.

4 . The system of claim 3 , wherein the feature vector is rotationally invariant along a single axis, the single axis being a gravitational axis.

5 . The system of claim 1 , wherein each feature vector comprises:
a first element indicative of the length of a segment defined by the respective scene point pair; a second element indicative of an angle between a segment defined by the respective scene point pair and a gravitational axis; a third element indicative of an angle that specifies an azimuth of the surface normal vector at a first point of the respective scene point pair; a fourth element indicative of an angle that specifies an elevation of the surface normal vector at the first point; a fifth element indicative of an angle that specifies an azimuth of the surface normal vector at a second point of the respective scene point pair; and a six element indicative of an angle that specifies an elevation of the surface normal vector at the second point.

6 . The system of claim 5 , wherein the angles of the third through sixth elements are defined with respect to a coordinate system uniquely determined by the scene point pair.

7 . The system of claim 1 , further comprising a k-dimensional tree in which the feature vector data of the plurality of models is stored.

8 . The system of claim 7 , wherein:
configuration instructions are stored in the memory; and the processor is configured via execution of the configuration instructions to adjust a distance value within which the feature vector data is searched to determine each nearest neighbor point pair.

9 . The system of claim 1 , wherein:
the processor is further configured via the execution of the feature vector instructions and/or the matching instructions to compute a canonical aligning transformation for each scene point pair of the set of point pairs based on oriented point pair data for each point pair; and the processor is further configured via the execution of the matching instructions to compute the respective aligning transformation for the nearest neighbor point pair by computing a rigid aligning transformation that aligns the respective scene point pair and each corresponding nearest neighbor point pair based on the canonical aligning transformation of the scene point pair.

10 . The system of claim 9 , wherein the rigid aligning transformation is further based on a pre-computed canonical aligning transformation for each nearest neighbor point pair of the pluralities of nearest neighbor point pairs, the pre-computed canonical aligning transformation being based on oriented point pair data for each nearest neighbor point pair.

11 . The system of claim 1 , wherein the aligning transformation specifies a six-dimensional transformation such that the model-transformation combination is a seven-dimensional data point.

12 . The system of claim 1 , wherein the aligning transformation specifies translation along three axes and rotation about the three axes.

13 . The system of claim 1 , wherein:
configuration instructions are stored in the memory; and the processor is configured via execution of the configuration instructions to establish the number of the plurality of nearest neighbor point pairs to be found for each scene point pair.

14 . An electronic device comprising:
a camera to capture input image data; a three-dimensional sensing system to generate a mesh from the input image data, the mesh comprising a number of data points and respective surface normal data for each data point; a model database in which feature vector data for a plurality of models is stored, the feature vector data of each model being indicative of a corresponding object of a plurality of objects; and an object detection system coupled to the model database, the object detection system configured to select a set of scene point pairs from the data points of the mesh, the object detection system comprising:
a feature vector calculator configured to determine a respective feature vector for each point pair of the set of scene point pairs based on the respective surface normal data for the data points of the respective scene point pair;
a matching engine configured to find, for each feature vector, a respective plurality of nearest neighbor point pairs in feature vector data of a plurality of models, the feature vector data of each model being indicative of a corresponding object of a plurality of objects, and further to compute, for each nearest neighbor point pair of the pluralities of nearest neighbor point pairs, a respective aligning transformation from the respective scene point pair to the nearest neighbor point pair, thereby defining a model-transformation combination for each nearest neighbor point pair, each model-transformation combination specifying the respective transformation and the respective model with which the nearest neighbor point pair is associated;
a voting engine configured to increment, with each binning of a respective one of the model-transformation combinations, a respective bin counter for the model-transformation combination, and further to select a number of the model-transformation combinations in accordance with the bin counters to detect a number of the objects in the scene input and estimate a pose of each detected object.

15 . The electronic device of claim 14 , wherein the feature vector is rotationally invariant al

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Sala Pablo
• Goussies Norberto

Classifications

• G06T2207/10028

  Range image; Depth image; 3D point clouds · CPC title

• G06T7/75 Primary

  involving models · CPC title

• G06T19/006

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G06F18/22

  Matching criteria, e.g. proximity measures · CPC title

• G06K9/52 Primary

  Physics · mapped topic