Contextual based image search results

Patent metadata

Field	Value
Publication number	US-11055335-B2
Application number	US-201615211535-A
Country	US
Kind code	B2
Filing date	Jul 15, 2016
Priority date	Jul 15, 2016
Publication date	Jul 6, 2021
Grant date	Jul 6, 2021

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium determining image relevance responsive to a search query. A method includes, for each resource in a plurality of resources, wherein each resource includes one or more images and text that is separate from each of the one or more images: determining, by a data processing apparatus, and from the text of the resource, resource topics described by the text of the resource. For each of the one or more images, processing the image to determine a set of image topics that describe topics to which content depicted in the image belongs. Determining, by the data process apparatus, one or more topic match scores, wherein each topic match score is a measure of relevance one or more of the image topics of the image to one or more of the resource topics of the resource.

First claim

What is claimed is:

1. A method performed by a data processing apparatus, comprising:
for each resource in a plurality of resources, wherein each resource includes one or more images and text that is separate from each of the one or more images, and each resource includes instructions that, when processed by one or more user devices, cause the one or more user devices to render, on a display of each of the one or more user devices, the resource, the one or more images included in the resource, and the text included in the resource:
determining, by a data processing apparatus, and from the text of the resource, resource topics described by the text of the resource;
for each image of the one or more images included in the resource:
processing, by the data processing apparatus, the image to determine a set of image topics that describe topics to which content depicted in the image belongs, wherein the determined image topics are not based on the text of the resource; and
determining, by the data processing apparatus, one or more topic match scores, wherein each topic match score is a measure of relevance of one or more of the image topics of the image to one or more of the resource topics of the resource that includes the image; and

storing in an index, by the data processing apparatus and for each image of the one or more images, data that describes the image topics for the image, resource topics for the resource, and the one or more topic match scores for the image.

2. The method of claim 1 , further comprising, for each image, processing, by the data processing apparatus, the image and the resource to determine one or more image prominence scores that collectively measure a prominence of how the image is displayed in the resource.

3. The method of claim 2 , wherein determining one or topic match scores comprises determining the one or more topic match scores based on measures of relevance of the image topics to the resource topics and the one or more image prominence scores.

4. The method of claim 3 , wherein determining the one or more topic match scores based on measures of relevance of the image topics to the resource topics and the one or more image prominence scores comprises adjusting the measures of relevance of the image topics to the resource topics by the one or more image prominence scores.

5. The method of claim 4 , wherein determining one or more prominence scores includes determining a prominence score based on an image size of the image.

6. The method of claim 4 , wherein determining one or more prominence scores includes determining a prominence score based on a display position of the image within the resource as rendered on a display of the user device.

7. The method of claim 1 , wherein:
processing the image to determine a set of image topics that describe topics to which content depicted in the image belongs comprises submitting the image to an image annotation system and receiving for the image a set of annotations as the image topics; and
the image annotation system determines the set of annotations based on computer image processing of the image that determines content depicted in the image, and the set of annotations are descriptive of the determined content.

8. The method of claim 1 , further comprising:
receiving an image-seeking query from a user device; and
determining, for each of the images, a search score that measures a relevance of the image to the image seeking query, the determining comprising, at least in part:
determining a relevance score that measures a relevance of the image-seeking query to the image topics of the image; and
determining, from the relevance score and the one or more topic match scores for the image stored in the index, the search score.

9. The method of claim 2 , further comprising:
receiving an image-seeking query from a user device; and
determining, for each of the images, a search score that measures a relevance of the image to the image seeking query, the determining comprising, at least in part:
determining a relevance score that measures a relevance of the image-seeking query to the image topics of the image; and
determining, from the relevance score, the prominence score, and the one or more topic match scores for the image stored in the index, the search score.

10. The method of claim 9 , further comprising:
ranking the images according the respective search scores for the images; and
providing, to a user device from which the image-seeking query was received, a set of image search results that identify a subset of the images, the subset selected according to the ranking.

11. The method of claim 1 , wherein processing the image to determine a set of image topics that describe topics to which content depicted in the image belongs is based on metadata tags associated with the image.

12. The method of claim 1 , wherein determining one or more topic match scores comprises determining the one or more topic match scores based on measures of relevance of the image topics to the resource topics in combination with a size of the image and a location of the image within the resource that includes the image.

13. The method of claim 1 , wherein determining one or more topic match scores comprises determining, as a topic match between the image and the resource, a similarity between the resource topics for the resource and the image topics for the image using a similarity function, wherein the image topics for the image include at least one topic different from each of the resource topics for the resource.

14. A system, comprising:
a data processing apparatus comprising one or more processors; and
software stored in non-transitory computer readable storage medium storing instructions executable by the data processing apparatus and that upon such execution cause the data processing apparatus to perform operations comprising:
for each resource in a plurality of resources, wherein each resource includes one or more images and text that is separate from each of the one or more images, and each resource includes instructions that, when processed by one or more user devices, cause the one or more user devices to render, on a display of each of the one or more user devices, the resource, the one or more images included in the resource, and the text included in the resource:
determining, by a data processing apparatus, and from the text of the resource, resource topics described by the text of the resource;
for each image of the one or more images included in the resource:
processing, by the data processing apparatus, the image to determine a set of image topics that describe topics to which content depicted in the image belongs, wherein the determined image topics are not based on the text of the resource; and
determining, by the data processing apparatus, one or topic match scores, wherein each topic match score is a measure of relevance of one or more of the image topics of the image to one or more of the resource topics of the resource that includes the image; and

storing in an index, by the data processing apparatus and for each image of the one or more images, data that describes the image topics for the image, resource topics for the resource, and the one or more topic match scores for the image.

15. The system of claim 14 , wherein the operations comprise, for each image, processing the image and the resource to determine one or more image prominence scores that collectively measure a prominence of how the image is displayed in the resource.

16. The system of claim 15 , wherein determining one or topic match sco

Assignees

• Google Llc

Inventors

• Zhang Wencong
• Sood Jyoti
• Tiwari Kamal
• Saikia Amitabh
• Manickavasagam Venkatesh
• Xiong Yilin

Classifications

• G06F16/951 Primary

  Indexing; Web crawling techniques · CPC title

• G06F16/51

  Indexing; Data structures therefor; Storage structures · CPC title

• G06F16/334

  Query execution (filtering based on additional data G06F16/335) · CPC title

• G06F16/35 Primary

  Clustering; Classification · CPC title

• G06F16/5866

  using information manually generated, e.g. tags, keywords, comments, manually generated location and time information · CPC title