Image-based retrieval and searching

Patent metadata

Field	Value
Publication number	US-9836481-B2
Application number	US-201514673523-A
Country	US
Kind code	B2
Filing date	Mar 30, 2015
Priority date	Mar 31, 2014
Publication date	Dec 5, 2017
Grant date	Dec 5, 2017

Abstract

In various example embodiments, a system and method for image query are presented. A feature detector generates first feature description data of a first type. An image processor accesses feature codes and angle bins. The image processor generates second feature description data of a second type by comparing a plurality of groups with respective codes of the feature codes. Each of the plurality of groups comprises at least of portion of the first feature description data determined based at least on the feature codes and the angle bins. A search engine selects a first one of a plurality of stored feature description data linked to respective stored images. An interface provides an indication of the stored image linked to the first stored feature description data for display of the stored image linked to the first stored feature description data on a remote device.

First claim

What is claimed is:

1. A system comprising:
a memory; and
one or more processors operatively coupled to the memory, the one or more processors being configured to cause the performance of operations comprising:
generate first feature description data of a first type from image data of an image query request provided by a remote device, the first feature description data comprising a first plurality of components and corresponding angles, each of the first plurality of components being indicative of an image feature of the image data, each of the corresponding angles representing an orientation of the image feature indicated by the corresponding component;
access feature codes and angle bins;
generate second feature description data of a second type by comparing a plurality of groups with respective codes of the feature codes, each of the plurality of groups comprising at least of portion of the first plurality of components that is determined based at least on comparing the first plurality of components with the feature codes and comparing the corresponding angles with the angle bins, wherein the second feature description data is generated by normalizing a vector that includes aggregations of residuals having a matching bin of the angle bins;
select a first one of a plurality of stored feature description data linked to respective stored images, the selecting of the first stored feature description data being based at least on comparing the second feature description data and the first stored feature description data; and
provide an indication of the stored image linked to the first stored feature description data for display of the stored image linked to the first stored feature description data on the remote device.

2. The system of claim 1 , wherein the feature detector module includes at least one of a Speeded Up Robust Feature (SURF) detector or a Scale-invariant Feature Transform (SIFT) detector to generate the generate the first feature description data from the image data of the image query request.

3. The system of claim 1 , wherein the corresponding angles represent angles in the image space of the image data of the image query request.

4. The system of claim 1 , wherein the second type corresponds to a Vector of Locally Aggregated Descriptors (VLAD) with aggregated descriptors for each pair of the feature codes and the angle bins.

5. The system of claim 1 , wherein the second feature description data comprise a second plurality of components, each of the second plurality of components corresponding to an aggregation of residuals of selected components of the first plurality of components with respect to a corresponding one of the feature codes, the selected components being selected based at least on comparing the first plurality of components with the feature codes and based at least on comparing the corresponding angles of the plurality of angles with the angle bins.

6. The system of claim 5 , wherein the comparing of the first plurality of components with the feature codes includes determining whether each component of the first plurality of components is closest to the corresponding one of the feature codes.

7. The system of claim 5 , wherein the second plurality of components corresponds to a product of a whitening matrix and the aggregation of residuals.

8. The system of claim 1 , wherein the search engine module is configured to select the first one of the plurality of stored feature description data linked in response to a determination that a distance between the second feature description data and the first one is less than a threshold.

9. A method comprising:
generating, by one or more processors, first feature description data of a first type from image data of an image query request provided by a remote device, the first feature description data comprising a first plurality of components and corresponding angles, each of the first plurality of components being indicative of an image feature of the image data, each of the corresponding angles representing an orientation of the image feature indicated by the corresponding component;
accessing feature codes and angle bins;
generating second feature description data of a second type by comparing a plurality of groups with respective codes of the feature codes, each of the plurality of groups comprising at least of portion of the first plurality of components that is determined based at least on comparing the first plurality of components with the feature codes and comparing the corresponding angles with the angle bins, wherein the second feature description data is generated by normalizing a vector that includes aggregations of residuals having a matching bin of the angle bins;
selecting a first one of a plurality of stored feature description data linked to respective stored images, the selecting of the first stored feature description data being based at least on comparing the second feature description data and the first stored feature description data; and
providing an indication of the stored image linked to the first stored feature description data for display of the stored image linked to the first stored feature description data on the remote device.

10. The method of claim 9 , wherein the first type corresponds to at least one of a Speeded Up Robust Feature (SURF) descriptor or a Scale-invariant Feature Transform (SIFT) descriptor.

11. The method of claim 9 , wherein the corresponding angles represent angles in the image space of the image data of the image query request.

12. The method of claim 9 , wherein the second type corresponds to a Vector of Locally Aggregated Descriptors (VLAD) with aggregated descriptors for each pair of the feature codes and the angle bins.

13. The method of claim 9 , wherein the second feature description data comprise a second plurality of components, each of the second plurality of components corresponding to an aggregation of residuals of selected components of the first plurality of components with respect to a corresponding one of the feature codes, the selected components being selected based at least on comparing the first plurality of components with the feature codes and based at least on comparing the corresponding angles of the plurality of angles with the angle bins.

14. A machine-readable medium embodying instructions that, in response to being executed by a device, cause the device to perform operations comprising:
generate first feature description data of a first type from image data of an image query request provided by a remote device, the first feature description data comprising a first plurality of components and corresponding angles, each of the first plurality of components being indicative of an image feature of the image data, each of the corresponding angles representing an orientation of the image feature indicated by the corresponding component;
access feature codes and angle bins;
generate second feature description data of a second type by comparing a plurality of groups with respective codes of the feature codes, each of the plurality of groups comprising at least of portion of the first plurality of components that is determined based at least on comparing the first plurality of components with the feature codes and comparing the corresponding angles with the angle bins, wherein the second feature description data is generated by normalizing a vector that includes aggregations of residuals having a matching bin of the angle bins;
select a first one of a plurality of stored feature description data linked to respective stored images, the selecting of the first stored feature description data being

Assignees

• Paypal Inc

Inventors

• Bhardwaj Anurag
• Di Wei
• Jagadeesh Vignesh
• Piramuthu Robinson
• Wang Zixuan

Classifications

• G06F16/53

  Querying · CPC title

• G06K9/52

  Physics · mapped topic

• G06F17/30244

  Physics · mapped topic

• G06F17/30247 Primary

  Physics · mapped topic

• G06K9/46

  Physics · mapped topic