Object recognition of feature-sparse or texture-limited subject matter

Patent metadata

Field	Value
Publication number	US-9720934-B1
Application number	US-201414209642-A
Country	US
Kind code	B1
Filing date	Mar 13, 2014
Priority date	Mar 13, 2014
Publication date	Aug 1, 2017
Grant date	Aug 1, 2017

Abstract

An object recognition system can be adapted to recognize subject matter having very few features or limited or no texture. A feature-sparse or texture-limited object can be recognized by complementing local features and/or texture features with color, region-based, shape-based, three-dimensional (3D), global, and/or composite features. Machine learning algorithms can be used to classify such objects, and image matching and verification can be adapted to the classification. Further, multiple modes of input can be integrated at various stages of the object recognition processing pipeline. These multi-modal inputs can include user feedback, additional images representing different perspectives of the object or specific regions of the object including a logo or text corresponding to the object, user behavior data, location, among others.

First claim

What is claimed is:

1. A non-transitory computer-readable storage medium storing instructions, the instructions, which when executed by one or more processors of one or more computing systems, cause the one or more computing systems to:
receive a request from a user to identify at least one query object represented in at least one query image;
determine a set of local features associated with the at least one query object, the set of local features comprising one or more image attributes, individual image attributes associated with a pixel region of the at least one query object, wherein the individual image attributes associated with the pixel region differ from the image attributes associated with a region immediately outside the pixel region;
determine whether the at least one query object is identifiable based on the set of local features;
in response to failing to identify the at least one query object based on the set of local features, determine a set of non-local features, individual non-local features comprising one of the one or more image attributes associated with a region that is larger than the pixel region associated with the individual image attributes for the set of local features for the at least one query object, each non-local feature being of a different type and including at least one of a color feature type, a region-based feature type, a shape-based feature type, a global feature type, a three-dimensional (3D) feature type, or a composite feature;
determine a classification of the at least one query object using a machine learning algorithm and the set of non-local features;
for each type of the set of non-local features, determine one or more database objects putatively matching the at least one query object using a respective similarity measure corresponding to the type, the one or more database objects corresponding to the classification;
determine, from among the one or more database objects putatively matching the at least one query object, at least one database object matching the at least query object using a respective geometric verification algorithm for each type of the set of non-local features;
retrieve information corresponding to the at least one database object; and
transmit the information corresponding to the at least one database object to the user.

2. The non-transitory computer-readable storage medium of claim 1 , wherein the instructions, which when executed by the one or more processors, further cause the one or more computing systems to:
transmit instructions for capturing at least one additional image including at least one portion of the at least one query object; and
receive and process the at least one additional image,
wherein the at least one database object matching the at least one query object is further based on the at least one additional image.

3. The non-transitory computer-readable storage medium of claim 2 , wherein the at least one additional image includes at least one of a different perspective of the at least one query object, a logo corresponding to the at least one query object, text corresponding to the at least one query object, or an indication of one or more dimensions of the at least one query object.

4. The non-transitory computer-readable storage medium of claim 1 , wherein the set of local features comprises points and edges of the at least one query object.

5. A computer-implemented method, comprising:
obtaining a request to identify at least one object represented in at least one image;
determining a set of features associated with the at least one object, the set of features including one or more local features and one or more non-local features, the one or more local features comprising one or more image attributes, individual image attributes associated with a pixel region of the at least one object, wherein the individual image attributes associated with the pixel region differ from the image attributes associated with a region immediately outside the pixel region, and the individual non-local features comprising one of the one or more image attributes associated with a region that is larger than the pixel region associated with the individual image attributes for the set of local features of the at least one object; and
in response to at least one of: (a) failing to identify the at least one object based on the one or more local features, (b) determining that a number of the one or more local features is below a local feature threshold value, (c) determining that respective extracted values of the one or more local features are below respective threshold values,
determining a classification of the at least one object based at least in part upon at least one portion of the one or more non-local features; and
determining at least one database object matching the at least one object based at least in part upon the classification.

6. The computer-implemented method of claim 5 , further comprising:
obtaining additional input data corresponding to a user associated with the request to identify the at least one object,
wherein determining the at least one database object matching the at least one object is further based at least in part upon the additional input data.

7. The computer-implemented method of claim 6 , wherein the additional input data includes at least one of a first image including a different perspective of the at least one object, a second image including a logo corresponding to the at least one object, a third image including text corresponding to the at least one object, a fourth image including an object providing an indication of one or more characteristics of the at least one object, user behavior data relating to the at least one object, or a location of the user.

8. The computer-implemented method of claim 5 , wherein the one or more non-local features include at least one of color features, region-based features, shape-based features, global features, 3-D features, or composite features.

9. The computer-implemented method of claim 5 , wherein determining the classification includes applying a machine learning algorithm using the at least one portion of the set of features.

10. The computer-implemented method of claim 9 , further comprising:
for each type of at least one second portion of the set of features, determining one or more database objects putatively matching the at least one object using a respective similarity measure corresponding to the type, the one or more database objects corresponding to the classification.

11. The computer-implemented method of claim 10 , wherein determining the at least one database object matching the at least one object is further based at least in part upon a respective geometric verification for each type of the at least one second portion of the set of features.

12. The computer-implemented method of claim 11 , wherein the respective geometric verification is based on a random sample consensus algorithm, and the method further comprising:
weighting, based at least in part upon the classification of the at least one object, a respective output of the respective geometric verification for each type of the at least one second portion of the set of features.

13. The computer-implemented method of claim 10 , wherein the similarity measure is based on one of a dot product, a Euclidean distance, Minkowski distance, a Mahalanobis distance, a quadratic form distance, Kullback-Leibler divergence, Jeffrey divergence, a Hausdorff distance, a Mallows distance, an earth mover's distance, an integrated region matching distance, or a machine learned rule.

14. The computer-implemented method of cla

Assignees

• A9 Com Inc

Inventors

• Dube Simant
• Hsiao Edward

Classifications

• G06F16/583 Primary

  using metadata automatically derived from the content · CPC title

• G06V10/75

  Organisation of the matching processes, e.g. simultaneous or sequential comparisons of image or video features; Coarse-fine approaches, e.g. multi-scale approaches; using context analysis; Selection of dictionaries · CPC title

• G06V10/764

  using classification, e.g. of video objects · CPC title

• G06F18/22

  Matching criteria, e.g. proximity measures · CPC title

• G06F18/24

  Classification techniques · CPC title