Contextual based image search results

Patent metadata

Field	Value
Publication number	US-2018018390-A1
Application number	US-201615211535-A
Country	US
Kind code	A1
Filing date	Jul 15, 2016
Priority date	Jul 15, 2016
Publication date	Jan 18, 2018
Grant date	—

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium determining image relevance responsive to a search query. In one aspect, a method includes, for each resource in a plurality of resources, wherein each resource includes one or more images and text that is separate from each of the one or more images: determining, by a data processing apparatus, and from the text of the resource, resource topics described by the text of the resource. For each of the one or more images, processing the image to determine a set of image topics that describe topics to which content depicted in the image belongs. Determining, by the data process apparatus, one or more topic match scores, wherein each topic match score is a measure of relevance one or more of the image topics of the image to one or more of the resource topics of the resource.

First claim

What is claimed is:

1 . A method performed by a data processing apparatus, comprising:
for each resource in a plurality of resources, wherein each resource includes one or more images and text that is separate from each of the one or more images:
determining, by a data processing apparatus, and from the text of the resource, resource topics described by the text of the resource;
for each of the one or more images,
processing, by the data processing apparatus, the image to determine a set of image topics that describe topics to which content depicted in the image belongs;
determining, by the data process apparatus, one or more topic match scores, wherein each topic match score is a measure of relevance of one or more of the image topics of the image to one or more of the resource topics of the resource;

storing in an index, by the data processing apparatus and for the image, data that describes the image topics for the image, resource topics for the resource, and the one or more topic match scores for the image.

2 . The method of claim 1 , further comprising, for each image, processing, by the data processing apparatus, the image and the resource to determine one or more image prominence scores that collectively measure a prominence of a display of the image in the resource.

3 . The method of claim 2 , wherein determining one or topic match scores comprises determining the one or more topic match scores based on measures of relevance of the image topics to the resource topics and the one or more image prominence scores.

4 . The method of claim 3 , wherein determining the one or more topic match scores based on measures of relevance of the image topics to the resource topics and the one or more image prominence scores comprises adjusting the measures of relevance of the image topics to the resource topics by the one or more prominence scores.

5 . The method of claim 4 , wherein determining one or more prominence scores includes determining a prominence score based on an image size of the image.

6 . The method of claim 4 , wherein determining one or more prominence scores includes determining a prominence score based on a position of the image within the resource as rendered on a user device.

7 . The method of claim 1 , wherein processing the image to determine a set of image topics that describe topics to which content depicted in the image belongs comprises submitting the image to an image annotation system and receiving for the image a set of annotations as the image topics.

8 . The method of claim 7 , wherein the image annotation system determines the labels based on computer image processing of the image that determines content depicted in the image, and the labels are descriptive of the determined content.

9 . The method of claim 1 , further comprising:
receiving an image-seeking query from a user device; determining, for each of the images, a search score that measures a relevance of the image to the image seeking query, the determining comprising, at least in part:
determining a relevance score that measures a relevance of the image-seeking query to the image topics of the image;
determining, from the relevance score and the one or more topic match scores for the image stored in the index, the search score.

10 . The method of claim 2 , further comprising:
receiving an image-seeking query from a user device; determining, for each of the images, a search score that measures a relevance of the image to the image seeking query, the determining comprising, at least in part:
determining a relevance score that measures a relevance of the image-seeking query to the image topics of the image;
determining, from the relevance score, the prominence score, and the one or more topic match scores for the image stored in the index, the search score.

11 . The method of claim 10 , further comprising:
ranking the images according the respective search scores for the images; and providing, to a user device from which the image-seeking query was received, a set of image search results that identify a subset of the images, the subset selected according to the ranking.

12 . A system, comprising:
a data processing apparatus; and software stored in non-transitory computer readable storage medium storing instructions executable by the data processing apparatus and that upon such execution cause the data processing apparatus to perform operations comprising:
for each resource in a plurality of resources, wherein each resource includes one or more images and text that is separate from each of the one or more images:
determining, by a data processing apparatus, and from the text of the resource, resource topics described by the text of the resource;
for each of the one the one or more images,
processing, by the data processing apparatus, the image to determine a set of image topics that describe topics to which content depicted in the image belongs;
determining, by the data process apparatus, one or topic match scores, wherein each topic match score is a measure of relevance of one or more of the image topics of the image to one or more of the resource topics of the resource;

storing in an index, by the data processing apparatus and for the image, data that describes the image topics for the image, resource topics for the resource, and the one or more topic match scores for the image.

13 . The system of claim 12 , further comprising, for each image, processing, by the data processing apparatus, the image and the resource to determine one or more image prominence scores that collectively measure a prominence of a display of the image in the resource.

14 . The system of claim 12 , wherein determining one or topic match scores comprises determining the one or more topic match scores based on measures of relevance of the image topics to the resource topics and the one or more image prominence scores.

15 . The system of claim 14 , wherein determining the one or more topic match scores based on measures of relevance of the image topics to the resource topics and the one or more image prominence scores comprises adjusting the measures of relevance of the image topics to the resource topics by the one or more prominence scores.

16 . The system of claim 15 , wherein determining one or more prominence scores includes determining a prominence score based on an image size of the image.

17 . The system of claim 15 , wherein determining one or more prominence scores includes determining a prominence score based on a position of the image within the resource as rendered on a user device.

18 . The system of claim 12 , further comprising:
receiving an image-seeking query from a user device; determining, for each of the images, a search score that measures a relevance of the image to the image seeking query, the determining comprising, at least in part:
determining a relevance score that measures a relevance of the image-seeking query to the image topics of the image;
determining, from the relevance score and the one or more topic match scores for the image stored in the index, the search score.

19 . The system of claim 13 , further comprising:
receiving an image-seeking query from a user device; determining, for each of the images, a search score that measures a relevance of the image to the image seeking query, the determining comprising, at least in part:
determining a relevance score that measures a relevance of the image-seeking query to the image topics of the image;
determining, from the

Assignees

• Google Inc

Inventors

• Zhang Wencong
• Sood Jyoti
• Tiwari Kamal
• Saikia Amitabh
• Manickavasagam Venkatesh
• Xiong Yilin

Classifications

• G06F16/35 Primary

  Clustering; Classification · CPC title

• G06F16/334

  Query execution (filtering based on additional data G06F16/335) · CPC title

• G06F16/51

  Indexing; Data structures therefor; Storage structures · CPC title

• G06F17/30268

  Physics · mapped topic

• G06F17/30705 Primary

  Physics · mapped topic