Method for object localization and pose estimation for an object of interest

Patent metadata

Field	Value
Publication number	US-9875427-B2
Application number	US-201514811062-A
Country	US
Kind code	B2
Filing date	Jul 28, 2015
Priority date	Jul 28, 2015
Publication date	Jan 23, 2018
Grant date	Jan 23, 2018

Abstract

A method for localizing and estimating a pose of a known object in a field of view of a vision system is described, and includes developing a processor-based model of the known object, capturing a bitmap image file including an image of the field of view including the known object, extracting features from the bitmap image file, matching the extracted features with features associated with the model of the known object, localizing an object in the bitmap image file based upon the extracted features, clustering the extracted features of the localized object, merging the clustered extracted features, detecting the known object in the field of view based upon a comparison of the merged clustered extracted features and the processor-based model of the known object, and estimating a pose of the detected known object in the field of view based upon the detecting of the known object.

First claim

The invention claimed is:

1. A method for localizing and estimating a pose of a known object in a field of view of a vision system, the known object including a structural entity having pre-defined features including spatial dimensions, the method comprising:
developing a processor-based model of the known object;
capturing a bitmap image file including an image of the field of view including the known object;
extracting features from the bitmap image file;
matching the extracted features with features associated with the model of the known object;
localizing an object in the bitmap image file based upon the extracted features, including identifying features in the bitmap image file associated with features of the known object, wherein identifying the features includes fitting a digital window around a region of interest in the bitmap image file and identifying features only in a portion of the bitmap image file within the digital window, and wherein fitting the digital window includes identifying inliers in the bitmap image file including data whose distribution can be explained by some set of model parameters associated with the known object;
clustering the extracted features of the localized object;
merging the clustered extracted features;
detecting the known object in the field of view based upon a comparison of the merged clustered extracted features and the processor-based model of the known object; and
estimating a pose of the detected known object in the field of view based upon the detecting of the known object.

2. The method of claim 1 , wherein extracting features from the bitmap image file includes employing a scale-invariant feature transform (SIFT) algorithm to detect distinctive image features from scale-invariant keypoints in the bitmap image file.

3. The method of claim 1 , wherein extracting features from the bitmap image file comprises extracting distinctive image features from scale-invariant keypoints in the bitmap image file based upon a correspondence between the extracted feature and an extracted feature in the processor-based model of the known object.

4. The method of claim 1 , wherein estimating a pose of the detected known object in the field of view based upon the detecting of the known object comprises executing a coarse-to-fine image matching step to detect a pose of the known object.

5. The method of claim 1 , wherein developing a processor-based model of the known object comprises:
capturing, employing a digital camera, digital images of the known object at a plurality of poses in relation to the digital camera;
executing feature tracking on the captured digital images;
building a three-dimensional (3D) point cloud associated with the poses based upon the feature tracking;
constructing a 3D mesh from the 3D point cloud; and
associating appearance descriptors with the 3D mesh.

6. The method of claim 5 , wherein capturing, employing a digital camera, digital images of the known object at a plurality of poses in relation to the digital camera comprises capturing video images of the known object at a plurality of poses; and
wherein executing feature tracking on the captured digital images comprises executing interframe feature tracking on the captured digital images.

7. A method for detecting a known object in a field of view of a vision system, the known object including a structural entity having pre-defined features including spatial dimensions, the method comprising:
developing a processor-based model of the known object;
capturing, via a single image detector, a bitmap image file including an image of the field of view including a known object;
extracting features from the bitmap image file;
matching the extracted features with features associated with the model of the known object;
localizing an object in the bitmap image file based upon the extracted features, including identifying features in the bitmap image file associated with features of the known object, wherein identifying the features includes fitting a digital window around a region of interest in the bitmap image file and identifying features only in a portion of the bitmap image file within the digital window, and wherein fitting the digital window includes identifying inliers in the bitmap image file including data whose distribution can be explained by some set of model parameters associated with the known object;
clustering the extracted features of the localized object;
merging the clustered extracted features; and
detecting the known object in the field of view based upon a comparison of the merged clustered extracted features and the processor-based model of the known object.

8. The method of claim 7 , wherein extracting features from the bitmap image file includes employing a scale-invariant feature transform (SIFT) algorithm to detect distinctive image features from scale-invariant keypoints in the bitmap image file.

9. The method of claim 7 , wherein extracting features from the bitmap image file comprises extracting distinctive image features from scale-invariant keypoints in the bitmap image file based upon a correspondence between the extracted feature and an extracted feature in the processor-based model of the known object.

10. The method of claim 7 , wherein developing a processor-based model of the known object comprises:
capturing, employing a digital camera, digital images of the known object at a plurality of poses in relation to the digital camera;
executing feature tracking on the captured digital images;
building a three-dimensional (3D) point cloud associated with the poses based upon the feature tracking;
constructing a 3D mesh from the 3D point cloud; and
associating appearance descriptors with the 3D mesh.

11. The method of claim 10 , wherein capturing, employing a digital camera, digital images of the known object at a plurality of poses in relation to the digital camera comprises capturing video images of the known object at a plurality of poses; and
wherein executing feature tracking on the captured digital images comprises executing interframe feature tracking on the captured digital images.

12. A method for determining a pose of an object of interest, comprising:
generating, by way of a digital camera, a three-dimensional (3D) digital image of a field of view;
executing object recognition in the digital image including detecting at least one recognized object;
extracting an object blob corresponding to the recognized object;
extracting a plurality of interest points from the object blob;
extracting a 3D point cloud and a 2D blob associated with the object blob;
comparing the interest points from the blob with interest points from each of a plurality of training images;
selecting one of the plurality of training images comprising the one of the training images having a greatest quantity of interest points similar to the interest points from the object blob;
saving the 3D point cloud and the 2D blob associated with the object blob;
calculating a rotation and a linear translation between the 3D point cloud associated with the object blob and the selected one of the training images employing an iterative closest point (ICP) algorithm; and
executing training to generate the plurality of training images, including:
capturing, using a digital camera, a plurality of training images of the known object at a plurality of different viewpoints;
converting each of the training images to bitmap image files;
extracting a main blob from each of the bitmap image files;
capturing features and interest points for the main blob;
extracting 3D points associated with the main blob; and
employing interpolation to

Assignees

• Gm Global Tech Operations Llc

Inventors

• Medasani Swarup
• Meltzer Jason
• Xu Jiejun
• Chen Zhichao
• Sundareswara Rashmi N
• Payton David W
• Uhlenbrock Ryan M
• Barajas Leandro G
• Kim Kyungnam

Classifications

• G06F18/23

  Clustering techniques · CPC title

• G06V10/462 Primary

  Salient features, e.g. scale invariant feature transforms [SIFT] · CPC title

• G06F18/22

  Matching criteria, e.g. proximity measures · CPC title

• G06F18/214

  Generating training patterns; Bootstrap methods, e.g. bagging or boosting · CPC title

• G06V20/653

  by matching three-dimensional models, e.g. conformal mapping of Riemann surfaces · CPC title