Global visual vocabulary, systems and methods

Patent metadata

Field	Value
Publication number	US-11170261-B2
Application number	US-201916732097-A
Country	US
Kind code	B2
Filing date	Dec 31, 2019
Priority date	Feb 13, 2014
Publication date	Nov 9, 2021
Grant date	Nov 9, 2021

Abstract

Systems and methods of generating a compact visual vocabulary are provided. Descriptor sets related to digital representations of objects are obtained, clustered and partitioned into cells of a descriptor space, and a representative descriptor and index are associated with each cell. Generated visual vocabularies could be stored in client-side devices and used to obtain content information related to objects of interest that are captured.

First claim

What is claimed is:

1. A computer-based method of instantiating an index-based image classifier system using at least one processor and at least one non-transitory computer readable medium, the method comprising:
obtaining a plurality of image characteristic sets associated with a plurality of digital representations of objects, each image characteristic existing within a characteristic space;
clustering the plurality of image characteristic sets into groups within the characteristic space;
partitioning the characteristic space into a plurality of partitioned regions based on the clustered groups;
assigning an index to each partitioned region based on the clustered groups;
instantiating an index-based image classifier system based on the assigned indices and image characteristic sets in each partitioned region such that the image classifier system is configured to generate a set of content indices that reference corresponding partitioned regions based on an input set of image characteristics;
receiving a query from a user device, wherein the user device is programmed to capture a digital representation of a scene including at least one object and extract at least one image characteristic set from the digital representation, the at least one image characteristic set existing within at least one characteristic space;
generating an input set of content indices based on the at least one image characteristic set extracted from the digital representation wherein the input set of content indices is refined by suppressing common content indices; and
retrieving at least one search result in response to the query from the user device targeting the image classifier system, wherein the query comprises the input set of content indices and one or more parameters including user device orientation, facing, and user device motion or heading, and the at least one search result comprises content information from a content database retrieved as a function of a match at a predetermined threshold level between the input set of content indices and a plurality of assigned indices associated with at least one known object.

2. The method of claim 1 , wherein the plurality of image characteristic sets comprises image descriptors.

3. The method of claim 1 , wherein the plurality of image characteristic sets comprises multi-modal descriptors.

4. The method of claim 1 , wherein the plurality of image characteristic sets comprises a homogenous mix of descriptors.

5. The method of claim 1 , wherein the image classifier system further comprises an invariant feature identification algorithm.

6. The method of claim 5 , wherein the invariant feature identification algorithm comprises at least one of a SIFT, FREAK, BRISK, and DAISY algorithm.

7. The method of claim 1 , wherein at least one of the plurality of image characteristic sets has its own descriptor space.

8. The method of claim 1 , wherein the at least one processor is further configured to cluster the plurality of image characteristic sets using at least one of hierarchal k-mean, approximate k-mean, k-means clustering, and histogram binning.

9. The method of claim 1 , wherein the at least one processor is further configured to partition the characteristic space based on Voronoi decomposition.

10. The method of claim 1 , wherein the image classifier system comprises a vocabulary tree.

11. The method of claim 10 , wherein the vocabulary tree comprises at least one of a k-nearest neighbor tree, a spill tree, and a k-d tree.

12. The method of claim 1 , wherein the image classifier system is further configured to generate the set of content indices using a nearest neighbor classification.

13. The method of claim 12 , wherein the image classifier system is further configured to calculate the nearest neighbor classification using at least one of a Euclidean distance and a Mahalanobis distance.

14. The method of claim 1 , wherein each one of the assigned indices is no more than six bytes.

15. The method of claim 14 , wherein each one of the assigned indices is no more than four bytes.

16. The method of claim 15 , wherein each one of the assigned indices is no more than three bytes.

17. The method of claim 1 , wherein the image classifier system is further configured to construct a query based on the input set of image characteristics.

18. The method of claim 1 , wherein the input set of image characteristics comprise image descriptors.

19. A non-transitory computer-readable medium having computer instructions stored thereon for instantiating an index-based image classifier system, which, when executed by a processor, cause the processor to perform one or more steps comprising:
obtaining a plurality of image characteristic sets associated with a plurality of digital representations of objects, each image characteristic existing within a characteristic space;
clustering the plurality of image characteristic sets into groups within the characteristic space;
partitioning the characteristic space into a plurality of partitioned regions based on the clustered groups;
assigning an index to each partitioned region based on the clustered groups;
instantiating an index-based image classifier system based on the assigned indices and image characteristic sets in each partitioned region such that the classifier system is configured to generate a set of content indices that reference corresponding partitioned regions based on an input set of image characteristics;
receiving a query from a user device, wherein the user device is programmed to capture a digital representation of a scene including at least one object and extract at least one image characteristic set from the digital representation, the at least one image characteristic set existing within at least one characteristic space;
generating an input set of content indices based on the at least one image characteristic set extracted from the digital representation wherein the input set of content indices is refined by suppressing common content indices; and
retrieving at least one search result in response to the query from the user device targeting the image classifier system, wherein the query comprises the input set of content indices and one or more parameters including user device orientation, facing, and user device motion or heading, and the at least one search result comprises content information from a content database retrieved as a function of a match at a predetermined threshold level between the input set of content indices and a plurality of assigned indices associated with at least one known object.

20. A system comprising:
one or more processors; and
at least one non-transitory memory device storing software instructions that, when executed by the one or more processors, cause the one or more processors to:
obtain a plurality of image characteristic sets associated with a plurality of digital representations of objects, each image characteristic existing within a characteristic space;
cluster the plurality of image characteristic sets into groups within the characteristic space;
partition the characteristic space into a plurality of partitioned regions based on the clustered groups;
assign an index to each partitioned region based on the clustered groups;

instantiate an index-based image classifier system based on the assigned indices and image characteristic sets in each partitioned region such that the classifier system is configured to generate a set of content indices that reference corresponding partitioned regions base

Assignees

• Nant Holdings Ip Llc

Inventors

• Song Bing
• Mckinnon David

Classifications

• G06V10/774

  Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting · CPC title

• G06F18/214

  Generating training patterns; Bootstrap methods, e.g. bagging or boosting · CPC title

• G06F40/242 Primary

  Dictionaries · CPC title

• G06F18/2163

  Partitioning the feature space · CPC title

• G06F18/24147

  Distances to closest patterns, e.g. nearest neighbour classification · CPC title