Organizing images associated with a user

Patent metadata

Field	Value
Publication number	US-11138476-B2
Application number	US-201916368561-A
Country	US
Kind code	B2
Filing date	Mar 28, 2019
Priority date	Dec 28, 2015
Publication date	Oct 5, 2021
Grant date	Oct 5, 2021

Abstract

A method includes identifying images associated with a user, where the image is identified as at least one of captured by a user device associated with the user, stored on the user device associated with the user, and stored in cloud storage associated with the user. The method also includes for each of the images, determining one or more labels, wherein the one or more labels are based on at least one of metadata and a primary annotation. The method also includes generating a mapping of the one or more labels to one or more confidence scores, wherein the one or more confidence scores indicate an extent to which the one or more labels apply to corresponding images. The method also includes interacting with the user to obtain identifying information that is used to categorize one or more of the images.

First claim

What is claimed is:

1. A computer-implemented method comprising:
identifying images associated with a user;
for each of the images, determining one or more labels, wherein the one or more labels are based on at least one of metadata, a primary annotation, or a secondary annotation;
generating a mapping of the one or more labels to one or more confidence scores, wherein the one or more confidence scores indicate an extent to which the one or more labels apply to a corresponding image;
receiving from the user a search query that includes two or more user-generated search terms for the images associated with the user, wherein the two or more user-generated search terms include a name of a place of interest and a proximity of the place of interest to a landmark;
translating each of the two or more user-generated search terms into two or more categorized search terms, wherein the categorized search terms correspond to latitude and longitude coordinates for a location based on the proximity of the place of interest to the landmark and a date, a holiday, a time, an altitude, or a direction;
identifying search results by determining a match between the two or more categorized search terms and the one or more labels in the mapping; and
ranking the search results based on the one or more confidence scores indicating the extent to which the one or more labels apply to each corresponding image.

2. The method of claim 1 , wherein the one or more labels are based on the primary annotation and further comprising:
generating the primary annotation by performing at least one of:
image recognition to determine one or more of an entity that appears in one or more of the images and a characteristic associated with one or more of the images, or
conversion of the metadata to the primary annotation based on an inference about the metadata.

3. The method of claim 1 , wherein:
the secondary annotation is generated by performing label expansion on at least one of the metadata or the primary annotation; and
the label expansion includes expanding the at least one of the metadata or the primary annotation based on a hierarchical taxonomy.

4. The method of claim 3 , wherein the label expansion includes expanding the at least one of the metadata or the primary annotation based on at least one of a semantic similarity of the at least one of the metadata or the primary annotation to the secondary annotation or a visual similarity of the at least one of the metadata or the primary annotation to the secondary annotation.

5. The method of claim 1 , further comprising:
generating, based on the two or more user-generated search terms and the mapping, one or more suggested search terms for the user that autocomplete the two or more user-generated search terms.

6. The method of claim 1 , wherein:
the two or more user-generated search terms further include a name of an event; and
the two or more categorized search terms include the date associated with the event.

7. The method of claim 1 , wherein the search query is a second search query and further comprising:
prior to receiving the second search query, receiving a first search query that includes the name of the place of interest;
identifying the search results from multiple matching places of interest; and
requesting the user to identify which of the multiple matching places of interest corresponds to the name of the place of interest.

8. The method of claim 1 , further comprising:
identifying a user activity to associate with the location; and
associating a user activity annotation with one or more of the images that are associated with the location.

9. The method of claim 1 , wherein the search query is a first search query and the search results are first search results, and further comprising:
receiving from the user a second search query;
determining, based on one or more terms in the second search query, that the second search query is to be applied to the first search results; and
providing the user with second search results that are filtered from the first search results and that match the second search query.

10. The method of claim 1 , wherein:
the mapping includes a graph of the images; and
the images represent nodes and each edge between the nodes is based on the one or more labels associated with corresponding images.

11. The method of claim 1 , wherein the two or more user-generated search terms further include a name of an event and the two or more categorized search terms include the date and the latitude and longitude coordinates associated with the event.

12. A computer system comprising:
one or more processors coupled to a memory;
an image processing module stored in the memory and executable by the one or more processors, the image processing module operable to identify images associated with a user and operable to, for each image, determine one or more labels for the image, wherein the one or more labels are based on at least one of metadata, a primary annotation, or a secondary annotation;
an indexing module stored in the memory and executable by the one or more processors, the indexing module operable to generate a mapping of the one or more labels to one or more confidence scores, wherein the one or more confidence scores indicate an extent to which the one or more labels apply to a corresponding image; and
a search module stored in the memory and executable by the one or more processors, the search module operable to:
receive from the user a search query that includes two or more user-generated search terms for the images associated with the user;
translate each of the two or more user-generated search terms into two or more categorized search terms, wherein the categorized search terms correspond to two or more of a date, a holiday, a time, latitude and longitude coordinates, an altitude, or a direction;
identify search results by determining a match between the two or more categorized search terms and the one or more labels in the mapping;
the search results based on the one or more confidence scores indicating the extent to which the one or more labels apply to each corresponding image;
identifying a user activity to associate with a location; and
associating a user activity annotation with one or more of the images that are associated with the location.

13. The system of claim 12 , wherein the search module is further operable to:
generate, based on the two or more user-generated search terms and the mapping, one or more suggested search terms for the user that autocomplete the two or more user-generated search terms.

14. The system of claim 12 , wherein the search module is further operable to:
retrieve additional information to translate the two or more user-generated search terms into the categorized search terms.

15. The system of claim 12 , wherein the search module is further operable to cluster the one or more of the images that are associated with the location based on the user activity.

16. A non-transitory computer storage medium encoded with a computer program, the computer program comprising instructions that, when executed by one or more computers, cause the one or more computers to perform operations comprising:
identifying images associated with a user;
for each of the images, determining one or more labels, wherein the one or more labels are based on a primary annotation,
wherein the primary annotation is generated by performing at least one of:
image recognition to determine one or more of an entity that appears in one of the images or a characteristic associated with one of t

Assignees

• Google Llc

Inventors

• Dua Robin
• Ravi Sujith

Classifications

• G06F16/58 Primary

  Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually · CPC title

• G06F18/24

  Classification techniques · CPC title

• G06F16/53

  Querying · CPC title

• G06F16/55

  Clustering; Classification · CPC title

• G06F16/50

  of still image data · CPC title