Generating labeled images

Patent metadata

Field	Value
Publication number	US-9256807-B1
Application number	US-201313803642-A
Country	US
Kind code	B1
Filing date	Mar 14, 2013
Priority date	Sep 27, 2012
Publication date	Feb 9, 2016
Grant date	Feb 9, 2016

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for generating labeled images. One of the methods includes selecting a plurality of candidate videos from videos identified in a response to a search query derived from a label for an object category; selecting one or more initial frames from each of the candidate videos; detecting one or more initial images of objects in the object category in the initial frames; for each initial frame including an initial image of an object in the object category, tracking the object through surrounding frames to identify additional images of the object; and selecting one or more images from the one or more initial images and one or more additional images as database images of objects belonging to the object category.

First claim

What is claimed is:

1. A method performed by one or more computers, the method comprising:
selecting a plurality of candidate videos from videos identified in a response to a search query derived from a label for an object category;
selecting one or more initial frames from each of the candidate videos;
detecting one or more initial images of objects in the object category in the initial frames;
for each initial frame including an initial image of an object in the object category, tracking the object through surrounding frames to identify additional images of the object;
selecting one or more images from the one or more initial images and one or more additional images as database images of objects belonging to the object category;
generating statistics that identify frequencies of co-occurrences of objects in the candidate videos; and
using the database images as training data for one or more learning models that predict a context of images or videos.

2. The method of claim 1 , further comprising:
storing the database images in association with a label for the object category.

3. The method of claim 1 , wherein detecting an initial image of an object in a particular initial frame comprises:
selecting a plurality of bounding boxes from the initial frame; and
selecting an image contained in a particular bounding box of the plurality of bounding boxes as an initial image of the object.

4. The method of claim 3 , wherein selecting the image contained in the particular bounding box of the plurality of bounding boxes as an initial image of the object comprises:
applying an object detector to each of the plurality of bounding boxes to generate a respective detection score for each of the bounding boxes; and
selecting a highest-scoring bounding box of the plurality of bounding boxes as containing an initial image of the object.

5. The method of claim 4 , further comprising:
determining that the detection score for the highest-scoring bounding box exceeds a detection score threshold value.

6. The method of claim 5 , further comprising:
adjusting the detection score threshold value based on a fraction of previously processed initial frames for which the highest-scoring bounding box has been found to satisfy the detection score threshold value.

7. The method of claim 1 , wherein tracking the object through surrounding frames to identify additional images of the object comprises:
tracking the object using an object tracker to identify bounding boxes; and
selecting images contained by one or more of the bounding boxes as additional images of the object.

8. The method of claim 1 , further comprising:
using the database images as training data for a particular object detector.

9. The method of claim 1 , further comprising:
using the database images as training data for a first learning model that takes as an input sequences of frames extracted from videos and predicts other frames in the videos.

10. The method of claim 1 , wherein the one or more learning models identify context terms for the videos or the images.

11. The method of claim 1 , further comprising:
using the database images as training data for a model of visual saliency.

12. A system comprising one or more computers and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform operations comprising:
selecting a plurality of candidate videos from videos identified in a response to a search query derived from a label for an object category;
selecting one or more initial frames from each of the candidate videos;
detecting one or more initial images of objects in the object category in the initial frames;
for each initial frame including an initial image of an object in the object category, tracking the object through surrounding frames to identify additional images of the object;
selecting one or more images from the one or more initial images and one or more additional images as database images of objects belonging to the object category;
generating statistics that identify frequencies of co-occurrences of objects in the candidate videos; and
using the database images as training data for one or more learning models that predict a context of images or videos.

13. The system of claim 12 , the operations further comprising:
storing the database images in association with a label for the object category.

14. The system of claim 12 , wherein detecting an initial image of an object in a particular initial frame comprises:
selecting a plurality of bounding boxes from the initial frame; and
selecting an image contained in a particular bounding box of the plurality of bounding boxes as an initial image of the object.

15. The system of claim 14 , wherein selecting the image contained in the particular bounding box of the plurality of bounding boxes as an initial image of the object comprises:
applying an object detector to each of the plurality of bounding boxes to generate a respective detection score for each of the bounding boxes; and
selecting a highest-scoring bounding box of the plurality of bounding boxes as containing an initial image of the object.

16. The system of claim 15 , the operations further comprising:
determining that the detection score for the highest-scoring bounding box exceeds a detection score threshold value.

17. The system of claim 16 , the operations further comprising:
adjusting the detection score threshold value based on a fraction of previously processed initial frames for which the highest-scoring bounding box has been found to satisfy the detection score threshold value.

18. The system of claim 12 , wherein tracking the object through surrounding frames to identify additional images of the object comprises:
tracking the object using an object tracker to identify bounding boxes; and
selecting images contained by one or more of the bounding boxes as additional images of the object.

19. The system of claim 12 , the operations further comprising:
using the database images as training data for a particular object detector.

20. The system of claim 12 , the operations further comprising:
using the database images as training data for a first learning model that takes as an input sequences of frames extracted from videos and predicts other frames in the videos.

21. The system of claim 12 , the wherein the one or more learning models identify context terms for the videos or the images.

22. The system of claim 12 , the operations further comprising:
using the database images as training data for a model of visual saliency.

23. A non-transitory computer-readable storage medium encoded with instructions that, when executed by one or more computers, cause the one or more computers to perform operations comprising:
selecting a plurality of candidate videos from videos identified in a response to a search query derived from a label for an object category;
selecting one or more initial frames from each of the candidate videos;
detecting one or more initial images of objects in the object category in the initial frames;
for each initial frame including an initial image of an object in the object category, tracking the object through surrounding frames to identify additional images of the object;
selecting one or more images from the one or more initial images and one or more additional images as database images of objects belonging to the obj

Assignees

• Google Inc

Inventors

• Shlens Jonathon
• Le Quoc V
• Corrado Gregory S
• Ranzato Marc Aurelio

Classifications

• G06F16/739 Primary

  in form of a video summary, e.g. the video summary being a video sequence, a composite still image or having synthesized frames · CPC title

• G06F18/2413

  based on distances to training or reference patterns · CPC title

• G06F18/28

  Determining representative reference patterns, e.g. by averaging or distorting; Generating dictionaries · CPC title

• G06F18/214

  Generating training patterns; Bootstrap methods, e.g. bagging or boosting · CPC title

• G06F18/22

  Matching criteria, e.g. proximity measures · CPC title