Refining image annotations

Patent metadata

Field	Value
Publication number	US-9727584-B2
Application number	US-201414498323-A
Country	US
Kind code	B2
Filing date	Sep 26, 2014
Priority date	May 30, 2012
Publication date	Aug 8, 2017
Grant date	Aug 8, 2017

Abstract

Methods, systems and apparatus for refining image annotations. In one aspect, a method includes receiving, for each image in a set of images, a corresponding set of labels determined to be indicative of subject matter of the image. For each label, one or more confidence values are determined. Each confidence value is a measure of confidence that the label accurately describes the subject matter of a threshold number of respective images to which it corresponds. Labels for which each of the one or more confidence values meets a respective confidence threshold are identified as high confidence labels. For each image in the set of images, labels in its corresponding set of labels that are high confidence labels are identified. Images having a corresponding set of labels that include at least a respective threshold number of high confidence labels are identified as high confidence images.

First claim

What is claimed is:

1. A method for automatically training an image relevance model and using the image relevance model to provide image search results in response to queries, the method being implemented by an image search apparatus comprising a data processing apparatus, and the method comprising:
receiving, by the image search apparatus and for each image in a set of images, a corresponding set of text labels, each text label being determined to be indicative of subject matter of the image;
for each text label, determining, by the image search apparatus, one or more confidence values, each confidence value being a measure of confidence that the text label accurately describes the subject matter of a threshold number of respective images to which the text label corresponds;
identifying, as high confidence labels and by the image search apparatus, text labels for which each of the one or more confidence values meets at least one of a precision measurement threshold and a frequency measurement threshold;
training, by the image search apparatus and using a set of training text labels and images corresponding to the training text labels, the image relevance model, wherein the trained image relevance model determines a relevance of an image to a text query, the set of training labels including only labels that have been identified as high confidence labels;
identifying, using the trained image relevance model, one or more images to provide in response to a received text query received from a user device; and
providing, in response to the received text query and to the user device, one or more search results that depict the one or more images.

2. The method of claim 1 , wherein determining one or more confidence values comprises determining a precision value for the text label, the precision value being a percentage of images that are determined to be relevant to the text label.

3. The method of claim 2 , wherein:
determining a precision value for the text label comprises:
determining, for the text label, selected images for which image search results were selected in response to a query that matches the text label; and
determining a percentage of the selected images that are included in the set of images and that each include the text label in its corresponding set of text labels; and

identifying text labels for which each of the one or more confidence values meets the precision measurement threshold comprises identifying text labels having precision values that meet the precision measurement threshold.

4. The method of claim 1 , wherein:
determining one or more confidence values comprises:
determining, for the text label, selected images for which image search results were selected in response to a query that matches the text label; and
determining a number of unique images in the selected images; and

identifying text labels for which each of the one or more confidence values meets the frequency measurement threshold comprises identifying text labels for which the number of unique images in the selected images meets a first frequency measurement threshold.

5. The method of claim 4 , wherein determining a number of unique images in the selected images comprises determining images for which image search results were selected in response to a query that matches the text label a minimum number of times.

6. The method of claim 1 , wherein:
determining one or more confidence values comprises determining, for the text label, a number of images in the set of images that each include the text label in its corresponding set of text labels; and
identifying text labels for which each of the one or more confidence values meets the frequency measurement threshold comprises identifying text labels for which the number of images in the set images meets a second frequency measurement threshold.

7. The method of claim 1 , wherein:
determining one or more confidence values comprises:
determining, for the text label, selected images for which image search results were selected in response to a query that matches the text label;
determining, for the text label, a precision value that is based on a percentage of the selected images that are included in the set of text images and that each include the text label in its corresponding set of text labels;
determining, for the text label, a number of unique images in the selected images; and
determining, for the text label, a number of images in the set of images that each include the text label in its corresponding set of text labels; and

identifying text labels for which each of the one or more confidence values meets at least one of a precision measurement threshold and a frequency measurement threshold as high confidence labels comprises:
identifying each text label having a precision value that meets a precision measurement threshold, and for which the number of unique images in the selected images meets a first frequency measurement threshold, and for which the number of images in the set images meets a second frequency measurement threshold as a high confidence label.

8. The method of claim 1 , wherein the sets of text labels corresponding to the images are determined according to the image relevance model, the image relevance model describing relationships between text labels and content feature values of images.

9. A system for automatically training an image relevance model and using the image relevance model to provide image search results in response to queries, the system comprising:
an image search apparatus comprising a data processing apparatus; and
a memory storage apparatus in data communication with the data processing apparatus, the memory storage apparatus storing instructions executable by the data processing apparatus and that upon such execution cause the data processing apparatus to perform operations comprising:
receiving, by the image search apparatus and for each image in a set of images, a corresponding set of text labels, each text label being determined to be indicative of subject matter of the image;
for each text label, determining, by the image search apparatus, one or more confidence values, each confidence value being a measure of confidence that the text label accurately describes the subject matter of a threshold number of respective images to which the text label corresponds;
identifying, as high confidence labels and by the image search apparatus, text labels for which each of the one or more confidence values meets at least one of a precision measurement threshold and a frequency measurement threshold;
training, by the image search apparatus and using a set of training text labels and images corresponding to the training text labels, the image relevance model, wherein the trained image relevance model determines a relevance of an image to a text query, the set of training labels including only labels that have been identified as high confidence labels;
identifying, using the trained image relevance model, one or more images to provide in response to a received text query received from a user device; and
providing, in response to the received text query and to the user device, one or more search results that depict the one or more images.

10. The system of claim 9 , wherein determining one or more confidence values comprises determining a precision value for the text label, the precision value being a percentage of images that are determined to be relevant to the text label.

11. The system of claim 10 , wherein:
determining a precision value for the text label comprises:
determining, for the text label, selected images for which image search results were selected in response to a query that matc

Assignees

• Google Inc

Inventors

• Alldrin Neil G
• Rosenberg Charles J
• Shen Bin
• Bengio Samy
• Zhou zhen hao

Classifications

• G06F16/5866 Primary

  using information manually generated, e.g. tags, keywords, comments, manually generated location and time information · CPC title

• G06K9/00664

  Physics · mapped topic

• G06F17/30268 Primary

  Physics · mapped topic

• G06V20/10 Primary

  Terrestrial scenes (scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59) · CPC title