System and method for product identification

Patent metadata

Field	Value
Publication number	US-9443164-B2
Application number	US-201414557677-A
Country	US
Kind code	B2
Filing date	Dec 2, 2014
Priority date	Dec 2, 2014
Publication date	Sep 13, 2016
Grant date	Sep 13, 2016

Abstract

A system and method for object instance localization in an image are disclosed. In the method, keypoints are detected in a target image and candidate regions are detected by matching the detected keypoints to keypoints detected in a set of reference images. Similarity measures between global descriptors computed for the located candidate regions and global descriptors for the reference images are computed and labels are assigned to at least some of the candidate regions based on the computed similarity measures. Performing the region detection based on keypoint matching while performing the labeling based on global descriptors improves object instance detection.

First claim

What is claimed is:

1. A method for object instance localization in an image comprising:
detecting keypoints in each of a set of labeled reference images, each of the reference images comprising an object instance;
describing each of the detected keypoints in the reference images with a local descriptor based on a region of pixels local to the respective keypoint;
receiving a target image;
detecting keypoints in the target image;
describing each of the detected keypoints in the target image with a local descriptor based on a region of pixels local to the respective keypoint;
locating candidate regions in the target image, based on matching of keypoints detected in the target image with keypoints detected in the reference images based on their local descriptors; and
computing similarity measures between global descriptors computed for the located candidate regions and global descriptors for the reference images; and
assigning labels to at least some of the candidate regions based on the computed similarity measures;
wherein at least one of the detecting keypoints, locating candidate regions, computing similarity measures, and the assigning of labels is performed with a processor.

2. The method of claim 1 , wherein the target image is of a product display unit which displays a collection of products and the object instances are products.

3. The method of claim 1 , further comprising computing a global descriptor for each of the at least some of the located candidate regions and for each reference image.

4. The method of claim 3 , wherein the computing of the global descriptors comprises for each of the candidate regions and reference images, extracting local descriptors from patches of the respective candidate region or reference image and generating a global descriptor having elements indicative of parameters of mixture model components of a mixture model representing the extracted local descriptors.

5. The method of claim 1 , wherein the global descriptors are Fisher Vectors.

6. The method of claim 1 , wherein the global descriptors are fixed length vectors.

7. The method of claim 1 , wherein the keypoint descriptors are gradient descriptors.

8. The method of claim 1 , wherein locating candidate regions comprises computing a Hough transform on subsets of the matched keypoints to generate the candidate regions.

9. The method of claim 1 , wherein each of the keypoint descriptors describes a local region of image pixels which contains fewer than one tenth of the image pixels in each candidate region.

10. The method of claim 1 , further comprising filtering the candidate regions to remove at least some overlapping candidate regions.

11. The method of claim 10 , wherein the filtering is performed by non-maximum suppression with a threshold overlap measure.

12. The method of claim 10 , wherein the computing of the global descriptors of the candidate regions is performed prior to the filtering of the candidate regions, the filtering being based on the computed similarity measures.

13. The method of claim 1 , wherein the detecting keypoints in the target image is performed without reference to the reference images.

14. A method for object instance localization in an image comprising:
detecting keypoints in each of a set of labeled reference images, each of the reference images comprising an object instance;
receiving a target image;
detecting keypoints in the target image, the detecting including at least one of Difference of Gaussians (DoG) detection, Hessian-Affine detection, corner detection, Harris-Affine detection, and extremal region detection;
locating candidate regions in the target image, based on matching of descriptors of the keypoints detected in the target image with descriptors of the keypoints detected in the reference images;
computing similarity measures between global descriptors computed for the located candidate regions and global descriptors for the reference images; and
assigning labels to at least some of the candidate regions based on the computed similarity measures;
wherein at least one of the detecting keypoints, locating candidate regions, computing similarity measures, and the assigning of labels is performed with a processor.

15. The method of claim 1 , further comprising outputting information based on the assigned labels.

16. A computer program product comprising a non-transitory recording medium storing instructions, which when executed on a computer causes the computer to perform the method of claim 1 .

17. A system comprising memory which stores instructions for performing the method of claim 1 and a processor in communication with the memory for executing the instructions.

18. A system for object instance localization in an image comprising:
memory which, for each of a set of labeled reference images, stores a global descriptor and a keypoint descriptor for each of a set of keypoints detected in the reference image, each of the reference images comprising an object instance;
a keypoint detection component which detects keypoints in a target image;
a keypoint description component which describes each of the detected keypoints in the target image with a local descriptor;
a keypoint matching component which matches keypoints in the target image to keypoints in the reference images based on their local descriptors;
a candidate region detector which locates candidate regions in the target image, based on the matched descriptors;
a feature extraction component which computes global descriptors for the located candidate regions;
a recognition component which computes similarity measures between the global descriptors computed for the located candidate regions and the global descriptors for the reference images and assigns labels to at least some of the candidate regions based on the computed similarity measures; and
a processor which implements the keypoint detection component, keypoint description component, keypoint matching component, candidate region detector, feature extraction component, and recognition component.

19. The system of claim 18 , further comprising a filtering component which filters the labeled candidate regions to remove at least some overlapping candidate regions.

20. A method comprising:
with a processor:
detecting keypoints in a target image based on the image content of the target image;
describing each of the detected keypoints with a local descriptor;
associating each of a set of labeled reference images with keypoints detected in the reference image, each of the reference images comprising an object instance;
matching keypoints in the target image to keypoints in the reference images based on their local descriptors;
locating candidate regions in the target image, based on the matched descriptors;
computing global descriptors for the located candidate regions;
computing similarity measures between the global descriptors computed for the located candidate regions and global descriptors computed for the reference images;
assigning labels to at least some of the candidate regions based on the computed similarity measures; and
outputting information based on the assigned labels.

Assignees

• Xerox Corp

Inventors

• Sulc Milan
• Gordo Soldevila Albert
• Larlus Larrondo Diane
• Perronnin Florent C

Classifications

• G06V20/20 Primary

  in augmented reality scenes · CPC title

• G06V10/25

  Determination of region of interest [ROI] or a volume of interest [VOI] · CPC title

• G06V10/757

  Matching configurations of points or features · CPC title

• G06V10/464

  using a plurality of salient features, e.g. bag-of-words [BoW] representations · CPC title

• G06T7/0046

  Physics · mapped topic