Method and system for generating a pictorial reference database using geographical information

Patent metadata

Field	Value
Publication number	US-9489402-B2
Application number	US-201414223819-A
Country	US
Kind code	B2
Filing date	Mar 24, 2014
Priority date	Jun 3, 2008
Publication date	Nov 8, 2016
Grant date	Nov 8, 2016

Abstract

For generating a pictorial reference database from a plurality of digital images, distributed geographical sub-areas are defined (S 1 ). For the geographical sub-areas, selected (S 2 ) from the plurality of digital images are images with a geo-tag located in the respective geographic sub-area. Feature vectors are generated (S 31 ) for localized visual features of the selected images. Based on the feature vectors, matching features are determined (S 32 ) for pairs of the selected images. For pairs with matching features, in each case, a measure of visual similarity is computed (S 33 ) based on different geometric arrangements of the localized visual features. Based on the visual similarity, clusters of images with matching objects are determined (S 5 ) and stored (S 6 ) in the pictorial reference database. The pictorial reference database can be generated automatically from a huge collection of images by dividing the images into geographical subsets which can be processed individually and possibly in parallel.

First claim

The invention claimed is:

1. A computer-implemented method of generating a pictorial reference database from a plurality of digital images, the method comprising:
defining a plurality of geographical sub-areas;
selecting digital images at least partially based on at least one of the plurality of geographical sub-areas, each of the selected digital images being associated with textual metadata information;
determining, for pairs of the selected images, a measure of textual similarity based on the textual metadata information associated with the selected images;
determining one or more links between pairs of the selected images, based upon the measures of textual similarity between the images;
determining clusters of images based on the one or more links;
associating a semantic interpretation of the images with the clusters of images; and
storing the clusters of images in the pictorial reference database.

2. The method of claim 1 , further comprising computing, for each pair of the selected images, a measure of visual similarity based on visual features of the respective selected images, wherein determining the clusters of images is based at least in part on the measures of visual similarity.

3. The method of claim 1 , further comprising classifying the clusters of images as an object type, indicative of a cluster comprising images of objects, or an event type, indicative of a cluster comprising images of events, based on at least one of time stamps of the images, number of different persons having provided the images, textual information of the images, or visual features of the images.

4. The method of claim 1 , further comprising:
determining, for each of the clusters of images, at least one link to information related to the respective cluster's images based on searching in one or more Internet-based data collection for word combinations occurring most frequently for the cluster; and
keeping from the search result links to pages which include images with at least a defined minimum similarity to images of the respective cluster.

5. The method of claim 1 , further comprising:
receiving, at an information server, an information request from a communication terminal, the information request including a digital image;
matching the received digital image to one or more matching images of one of the clusters stored in the reference database; and
returning to the communication terminal cluster information containing the matching images, the cluster information including at least one of textual information, geographical location information or a link for retrieving further related information.

6. The method of claim 1 , further comprising determining, for images of a cluster, at least one user-selectable area in the respective image for activating a link to information of the cluster, based at least in part on locations of matching features having a defined level of occurrence frequency in the respective cluster.

7. A computer program product comprising a non-transitory computer readable medium having stored thereon computer program code to:
define a plurality of geographical sub-areas;
select, from a plurality of digital images, a plurality of images at least partially based on at least one of the plurality of geographical sub-areas, each of the selected images being associated with textual metadata information;
determine, for pairs of the selected images, a measure of textual similarity at least partially based on textual metadata information associated with the selected images;
determine one or more links between pairs of the selected images, based upon the measures of textual similarity between the images;
determine clusters of images with matching objects based on the one or more links;
associate a semantic interpretation of the images with the clusters of images; and
store the clusters of images in a pictorial reference database.

8. The computer program product of claim 7 , wherein the non-transitory computer readable medium has stored thereon computer program code to:
compute, for pairs of the selected images a measure of visual similarity based on visual features of the respective selected images; and
determine the clusters of images with matching objects based at least in part on the measures of visual similarity.

9. The computer program product of claim 7 , wherein the non-transitory computer readable medium has stored thereon computer program code to classify the clusters of images as an object type, indicative of a cluster comprising images of objects, or an event type, indicative of a cluster comprising images of events, based on at least one of time stamps of the images, number of different persons having provided the images, textual information associated with the images, or visual features of the images.

10. The computer program product of claim 7 , wherein the non-transitory computer readable medium has stored thereon computer program code to:
determine, for each of the clusters of images, at least one link to information related to the respective cluster's images based on searching in one or more Internet-based data collection for word combinations occurring most frequently for the cluster; and
keep from the search result links to pages which include images with at least a defined minimum similarity to images of the respective cluster.

11. The computer program product of claim 7 , wherein the non-transitory computer readable medium has stored thereon computer program code to:
receive, at an information server, an information request from a communication terminal, the information request including a digital image;
match the received digital image to one or more matching images of one of the clusters stored in the reference database; and
return to the communication terminal cluster information containing the matching images, the cluster information including at least one of textual information, geographical location information or a link for retrieving further related information.

12. The computer program product of claim 7 , wherein the non-transitory computer readable medium has stored thereon computer program code to determine, for images of a cluster, at least one user-selectable area in the respective image for activating a link to information of the cluster, based at least in part on locations of matching features having a defined level of occurrence frequency in the respective cluster.

13. A server for generating a pictorial reference database from a plurality of digital images, the system comprising:
a memory configured to store a definition of a plurality of geographical sub-areas, and to store clusters of images in the pictorial reference database;
a processor configured to:
select, from the digital images, digital images at least partially based on at least one of the plurality of geographical sub-areas, each of the selected digital images being associated with textual metadata information;
determine, for pairs of the selected images, a measure of textual similarity based on textual metadata information associated with the selected images;
determine one or more links between pairs of the selected images, based upon the measures of textual similarity between the images;
determine clusters of images based on the one or more links; and
associated a semantic interpretation of the images with the clusters of images.

14. The server of claim 13 , wherein the processor is further configured to compute, for pairs of the selected images, a measure of visual similarity based on visual features of the respective selected images, wherein the clusters of images are determined based upon mat

Assignees

• Qualcomm Inc

Inventors

• Quack Till

Classifications

• G06V10/24

  Aligning, centring, orientation detection or correction of the image · CPC title

• G06F17/30241

  Physics · mapped topic

• G06T17/05 Primary

  Geographic models · CPC title

• G06K9/32

  Physics · mapped topic

• G06K9/00664

  Physics · mapped topic