Systems, methods, and apparatus for image retrieval

Patent metadata

Field	Value
Publication number	US-10089330-B2
Application number	US-201414576006-A
Country	US
Kind code	B2
Filing date	Dec 18, 2014
Priority date	Dec 20, 2013
Publication date	Oct 2, 2018
Grant date	Oct 2, 2018

Abstract

A method of image retrieval includes obtaining information identifying a plurality of selected objects and selecting one among a plurality of candidate geometrical arrangements. This method also includes, by at least one processor, and in response to the selecting, identifying at least one digital image, among a plurality of digital images, that depicts the plurality of selected objects arranged according to the selected candidate geometrical arrangement.

First claim

What is claimed is:

1. A method of image retrieval, the method comprising:
obtaining, at a processor, information selecting a plurality of objects represented in a digital image, the digital image associated with a location space and the plurality of selected objects distributed about the location space;
selecting, by the processor, a particular candidate geometrical arrangement of the plurality of selected objects among multiple candidate geometrical arrangements of the plurality of selected objects;
determining, at the processor, a reference position based on the particular candidate geometrical arrangement of the plurality of selected objects;
generating, by the processor, metadata descriptive of the particular candidate geometrical arrangement and the reference position; and
searching, by the processor, a metadata index corresponding to a plurality of digital images to identify at least one digital image from the plurality of digital images, the at least one digital image identified based on comparison of the metadata with an entry of the metadata index such that the at least one digital image depicts the plurality of selected objects, relative to the reference position, arranged according to the particular candidate geometrical arrangement.

2. The method of image retrieval according to claim 1 , wherein the obtained information includes, for each of the plurality of selected objects, a label that is associated with the object, wherein the searching includes searching for the labels within the metadata index that is associated with the plurality of digital images.

3. The method of image retrieval according to claim 1 , wherein the searching comprises searching, within metadata index that is associated with the plurality of digital images, for a label that is associated with the particular candidate geometrical arrangement of the plurality of selected objects, and wherein the reference position is one of a center of mass of the particular candidate geometrical arrangement of the plurality of selected objects, a top pixel or a bottom pixel among the plurality of selected objects, a vertical projection of the center of mass of the particular candidate geometrical arrangement of the plurality of selected objects to a top of the particular candidate geometrical arrangement of the plurality of selected objects or a bottom of the particular candidate geometrical arrangement of the plurality of selected objects, an average of positions of each of the plurality of selected objects of the particular candidate geometrical arrangement of the plurality of selected objects, or a position of a particular one of the plurality of selected objects.

4. The method of image retrieval according to claim 1 , wherein the at least one digital image includes a plurality of frames from a first video file and a plurality of frames from a second video file.

5. The method of image retrieval according to claim 1 , wherein the obtaining information selecting the plurality of selected objects is performed using at least one of speech recognition and a touchscreen, the speech recognition associated with at least one of:
a name of an object of the plurality of selected objects,
a particular arrangement of the plurality of selected objects using descriptors, and

the touchscreen configured to enable one of:
selection of the object via an associated icon,
highlighting the object within the digital image, and
surrounding at least a portion of the object within a bounding box, and
wherein the digital image is associated with one or more of a sporting events, a social events, an art performance, and security or surveillance monitoring.

6. The method of image retrieval according to claim 1 , wherein a selection of the plurality of selected objects is based upon one of a bounding box, a bounding ellipse, and a lasso being placed around each object or a portion of each object of the plurality of selected objects, wherein the one of the bounding box, the bounding ellipse, and the lasso indicates a location and a size within the location space.

7. The method of image retrieval according to claim 1 , wherein reference position corresponds to:
a center of mass of an arrangement of the plurality of selected objects,
a top pixel representing one of the plurality of selected objects,
a bottom pixel representing one of the plurality of selected objects, or
a vertical projection of the center of mass.

8. The method of image retrieval according to claim 1 , wherein reference position is based on a position of a particular one of the plurality of selected objects or an average of positions of the plurality of selected objects in a ground plane, and wherein the metadata is based on an approximation of a count of the plurality of selected objects.

9. The method of image retrieval according to claim 1 , further comprising receiving, at the processor, a set of multiple candidate geometrical arrangements of the plurality of selected objects, wherein the location space is divided into a plurality of regions, and wherein the particular candidate geometrical arrangement is selected based on a ratio between a first count of one or more particular objects in a first region of the location space and a second count of the one or more particular objects in a second region of the location space.

10. The method of image retrieval according to claim 1 , further comprising:
clustering, by the processor, two or more frames having geometric arrangements of the plurality of selected objects that are similar to the particular candidate geometric arrangement; and
generating, by the processor, metadata descriptive of a cluster of the two or more frames.

11. The method of image retrieval according to claim 10 , wherein the location space is divided into a plurality of regions, wherein a codebook corresponds to a division scheme associated with the location space, and wherein the metadata is associated with multiple codebooks.

12. The method of image retrieval according to claim 1 , wherein the plurality of selected objects includes a first object having first coordinate data, a second object having second coordinate data, and a third object having third coordinate data, and wherein generating the metadata comprises:
mapping the first coordinate data to a first codeword associated with a codebook, the first codeword representing a first region of the location space;
mapping the second coordinate data to a second codeword associated with the codebook, the second codeword representing a second region of the location space; and
mapping the third coordinate data to a third codeword associated with the codebook, the third codeword representing a third region of the location space,
wherein the metadata includes the first codeword, the second codeword, and the third codeword.

13. The method of image retrieval according to claim 12 :
wherein the metadata indicates a region of the location space which includes the reference position;
wherein the location space is one of a pixel coordinate space, a ground plane of a scene space, a two-dimensional space, and a three-dimensional space;
wherein the first region, the second region, and the third region divide the location space into regions of unequal size, the first region having a high density of the plurality of selected objects and the second region having a low density of the plurality of selected objects; and
wherein the first region is smaller than the smaller region, and more codewords are associated with the first region than the second region.

14. A non-transitory computer-readable medium storing instructions that when executed by a

Assignees

• Qualcomm Inc

Inventors

• Gao Dashan
• Zhong Xin
• Kandhadai Ananthapadmanabhan Arasanipalai
• Laver Steven Douglas

Classifications

• G06F40/279

  Recognition of textual entities · CPC title

• G06F16/35

  Clustering; Classification · CPC title

• G06F16/9535

  Search customisation based on user profiles and personalisation · CPC title

• G06F16/7335

  Graphical querying, e.g. query-by-region, query-by-sketch, query-by-trajectory, GUIs for designating a person/face/object as a query predicate (end-user interface involving hot spots associated with the video H04N21/4725; end-user interface for selecting a Region of Interest H04N21/4728) · CPC title

• G06T11/60

  Creating or editing images; Combining images with text · CPC title