Real-time, model-based object detection and pose estimation

Patent metadata

Field	Value
Publication number	US-10062003-B2
Application number	US-201715812539-A
Country	US
Kind code	B2
Filing date	Nov 14, 2017
Priority date	Jun 24, 2015
Publication date	Aug 28, 2018
Grant date	Aug 28, 2018

Abstract

A system includes a memory and a processor configured to select a set of scene point pairs, to determine a respective feature vector for each scene point pair, to find, for each feature vector, a respective plurality of nearest neighbor point pairs in feature vector data of a number of models, to compute, for each nearest neighbor point pair, a respective aligning transformation from the respective scene point pair to the nearest neighbor point pair, thereby defining a respective model-transformation combination for each nearest neighbor point pair, each model-transformation combination specifying the respective aligning transformation and the respective model with which the nearest neighbor point pair is associated, to increment, with each binning of a respective one of the model-transformation combinations, a respective bin counter, and to select one of the model-transformation combinations in accordance with the bin counters to detect an object and estimate a pose of the object.

First claim

What is claimed is:

1. A system comprising:
a memory in which feature vector instructions, matching instructions, and voting instructions are stored; and
a processor coupled to the memory, the processor configured via execution of:
the feature vector instructions to select a set of scene point pairs of a scene input, and to determine a respective feature vector for each scene point pair of the set of scene point pairs;
the matching instructions to find, for each feature vector, a respective plurality of nearest neighbor point pairs in feature vector data of a plurality of models, the feature vector data of each model being indicative of a corresponding object of a plurality of objects, and further to compute, for each nearest neighbor point pair of the pluralities of nearest neighbor point pairs, a respective aligning transformation from the respective scene point pair to the nearest neighbor point pair, thereby defining a respective model-transformation combination for each nearest neighbor point pair, each model-transformation combination specifying the respective model with which the nearest neighbor point pair is associated; and
the voting instructions to detect an object in the scene input and estimate a pose of the detected object by accumulating votes for the model-transformation combinations.

2. The system of claim 1 , wherein the processor is further configured via the execution of the matching instructions to:
access a search tree database in which the feature vector data is stored; and
query the search tree database to determine the plurality of nearest neighbor point pairs based on a respective distance between the feature vector of the respective scene point pair and the feature vector data of each model.

3. The system of claim 1 , wherein the feature vector is descriptive of local geometry at each point of the scene point pair in a translationally and rotationally invariant manner.

4. The system of claim 3 , wherein the feature vector is rotationally invariant along a single axis, the single axis being a gravitational axis.

5. The system of claim 1 , wherein each feature vector comprises:
a first element indicative of the length of a segment defined by the respective scene point pair;
a second element indicative of an angle between a segment defined by the respective scene point pair and a gravitational axis;
a third element indicative of an angle that specifies an azimuth of the surface normal vector at a first point of the respective scene point pair;
a fourth element indicative of an angle that specifies an elevation of the surface normal vector at the first point;
a fifth element indicative of an angle that specifies an azimuth of the surface normal vector at a second point of the respective scene point pair; and
a six element indicative of an angle that specifies an elevation of the surface normal vector at the second point.

6. The system of claim 5 , wherein the angles of the third through sixth elements are defined with respect to a coordinate system uniquely determined by the scene point pair.

7. The system of claim 1 , further comprising a k-dimensional tree in which the feature vector data of the plurality of models is stored.

8. The system of claim 7 , wherein:
configuration instructions are stored in the memory; and
the processor is configured via execution of the configuration instructions to adjust a distance value within which the feature vector data is searched to determine each nearest neighbor point pair.

9. The system of claim 1 , wherein:
the processor is further configured via the execution of the feature vector instructions and/or the matching instructions to compute a canonical aligning transformation for each scene point pair of the set of point pairs based on oriented point pair data for each point pair; and
the processor is further configured via the execution of the matching instructions to compute the respective aligning transformation for the nearest neighbor point pair by computing a rigid aligning transformation that aligns the respective scene point pair and each corresponding nearest neighbor point pair based on the canonical aligning transformation of the scene point pair.

10. The system of claim 9 , wherein the rigid aligning transformation is further based on a pre-computed canonical aligning transformation for each nearest neighbor point pair of the pluralities of nearest neighbor point pairs, the pre-computed canonical aligning transformation being based on oriented point pair data for each nearest neighbor point pair.

11. The system of claim 1 , wherein the aligning transformation specifies a six-dimensional transformation such that the model-transformation combination is a seven-dimensional data point.

12. The system of claim 1 , wherein the aligning transformation specifies translation along three axes and rotation about the three axes.

13. The system of claim 1 , wherein:
configuration instructions are stored in the memory; and
the processor is configured via execution of the configuration instructions to establish the number of the plurality of nearest neighbor point pairs to be found for each scene point pair.

14. An electronic device comprising:
a display;
a processor coupled to the display; and
one or more computer-readable storage media in which computer-readable instructions are stored such that, when executed by the processor, direct the processor to:
select a set of scene point pairs of a scene input;
determine a respective feature vector for each scene point pair of the set of scene point pairs;
find, for each feature vector, a respective plurality of nearest neighbor point pairs in feature vector data of a number of models, the feature vector data of each model being indicative of a corresponding object of a plurality of objects;
compute, for each nearest neighbor point pair of the pluralities of nearest neighbor point pairs, a respective aligning transformation from the respective scene point pair to the nearest neighbor point pair, thereby defining a model-transformation combination for each nearest neighbor point pair, each model-transformation combination specifying the respective model with which the nearest neighbor point pair is associated;
detect a number of the objects in the scene input and estimate a pose of each detected object by accumulating votes for the model-transformation combinations; and
modify output data generated to direct rendering of images on the display in accordance with each detected object and the estimated pose of each detected object.

15. The electronic device of claim 14 , wherein the processor is further directed by the computer-readable instructions to:
compute a canonical aligning transformation for each scene point pair of the set of point pairs based on oriented point pair data for each point pair; and
compute the respective aligning transformation for the nearest neighbor point pair by computing a rigid aligning transformation that aligns the respective scene point pair and each corresponding nearest neighbor point pair based on the canonical aligning transformation of the scene point pair.

16. The electronic device of claim 15 , wherein the rigid aligning transformation is further based on a pre-computed canonical aligning transformation for each nearest neighbor point pair of the pluralities of nearest neighbor point pairs based on oriented point pair data for each nearest neighbor point pair.

17. A method comprising:
selecting a set of scene point pairs of a scene input;
determining a respective feature vector for each scene point

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Sala Pablo
• Goussies Norberto

Classifications

• G06T2207/10028

  Range image; Depth image; 3D point clouds · CPC title

• G06T7/75 Primary

  involving models · CPC title

• G06T19/006

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G06F18/22

  Matching criteria, e.g. proximity measures · CPC title

• G06K9/6201

  Physics · mapped topic