Generating labeled images

Patent metadata

Field	Value
Publication number	US-9852363-B1
Application number	US-201614987955-A
Country	US
Kind code	B1
Filing date	Jan 5, 2016
Priority date	Sep 27, 2012
Publication date	Dec 26, 2017
Grant date	Dec 26, 2017

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for generating labeled images. One of the methods includes selecting a plurality of candidate videos from videos identified in a response to a search query derived from a label for an object category; selecting one or more initial frames from each of the candidate videos; detecting one or more initial images of objects in the object category in the initial frames; for each initial frame including an initial image of an object in the object category, tracking the object through surrounding frames to identify additional images of the object; and selecting one or more images from the one or more initial images and one or more additional images as database images of objects belonging to the object category.

First claim

What is claimed is:

1. A method of generating labeled images of objects belonging to a particular object category, the method comprising:
obtaining data identifying the particular object category;
identifying database images of the objects belonging to the particular object category, comprising:
generating a search query derived from a label for the particular object category;
selecting a plurality of candidate videos from videos identified in response to submission of the search query;
selecting one or more initial frames from each of the candidate videos;
detecting one or more initial images of objects belonging to the particular object category in the one or more initial frames;
for each initial frame including an initial image of an object belonging to the particular object category, tracking the object through surrounding frames to identify one or more additional images of the object; and
selecting one or more images from the one or more initial images and one or more additional images as the database images of objects belonging to the particular object category; and

storing the database images as images of objects belong to the particular object category.

2. The method of claim 1 , wherein storing the database images comprises storing the database images as training data for a machine learning model.

3. The method of claim 2 , further comprising:
training, using the training data that includes the database images, the machine learning model to classify particular images in videos as being associated with the object category.

4. The method of claim 2 , further comprising:
training, using the training data that includes the database images, the machine learning model to predict a context of images or videos.

5. The method of claim 2 , further comprising:
training, using the training data that includes the database images, the machine learning model to receive, as an input, sequences of frames extracted from videos and predict other frames in the videos.

6. The method of claim 1 , further comprising:
generating an additional search query derived from terms associated with the label for the particular object category;
identifying one or more videos identified in response to submission of the additional search query; and
merging the plurality of candidate videos and the one or more videos identified in response to submission of the additional search query to generate a second plurality of candidate videos.

7. The method of claim 6 , wherein merging the plurality of candidate videos and the one or more videos comprises:
determining candidate videos that were included in both (i) the one or more videos identified in response to submission of the additional search query, and (ii) the videos identified in response to submission of the search query; and
generating the second plurality of candidate videos by filtering videos from the plurality of candidate videos that were not included in both (i) the one or more videos identified in response to submission of the additional search query, and (ii) the videos identified in response to submission of the search query.

8. The method of claim 1 , wherein selecting the one or more images from the one or more initial images and the one or more additional images as the database images of objects belonging to the particular object category comprises:
determining one or more subsets of the one or more initial images in the respective selected candidate video; and
selecting subsets of the one or more initial images that (i) are spaced apart from each other by a determined number of images in the respective selected candidate videos, and (ii) satisfy a quality threshold for initial images.

9. The method of claim 1 , wherein for each initial frame including the initial image of the object belonging to the particular object category, tracking the object through surrounding frames to identify one or more additional images of the object comprises:
applying an object tracker to a plurality of bounding boxes within the initial frame to track the initial image of the object belonging to the particular object category;
generating scores for the plurality of bounding boxes in response to applying the object tracker;
determining a detection score threshold value based on a fraction of previously-processed initial frames for which the highest-scoring bounding box has been found to satisfy a previous detection score threshold value being above a first threshold fraction; and
selecting a particular bounding box of the plurality of bounding boxes having the highest score and satisfying the detection score threshold.

10. A system comprising:
one or more processors and one or more computer storage media storing instructions that are operable and when executed by the one or more processors, cause the one or more processors to perform operations comprising:
obtaining data identifying the particular object category;
identifying database images of the objects belonging to the particular object category, comprising:
generating a search query derived from a label for the particular object category;
selecting a plurality of candidate videos from videos identified in response to submission of the search query;
selecting one or more initial frames from each of the candidate videos;
detecting one or more initial images of objects belonging to the particular object category in the one or more initial frames;
for each initial frame including an initial image of an object belonging to the particular object category, tracking the object through surrounding frames to identify one or more additional images of the object; and
selecting one or more images from the one or more initial images and one or more additional images as the database images of objects belonging to the particular object category; and

storing the database images as images of objects belong to the particular object category.

11. The system of claim 10 , wherein storing the database images comprises storing the database images as training data for a machine learning model.

12. The system of claim 11 , wherein the operations further comprise training, using the training data that includes the database images, the machine learning model to:
classify particular images in videos as being associated with the object category;
predict a context of images or videos; or
receive, as an input, sequences of frames extracted from videos and predict other frames in the videos.

13. The system of claim 10 , wherein the operations further comprise:
generating an additional search query derived from terms associated with the label for the particular object category;
identifying one or more videos identified in response to submission of the additional search query; and
merging the plurality of candidate videos and the one or more videos identified in response to submission of the additional search query to generate a second plurality of candidate videos.

14. The system of claim 13 , wherein merging the plurality of candidate videos and the one or more videos comprises:
determining candidate videos that were included in both (i) the one or more videos identified in response to submission of the additional search query, and (ii) the videos identified in response to submission of the search query; and
generating the second plurality of candidate videos by filtering videos from the plurality of candidate videos that were not included in both (i) the one or more videos identified in response to submission of the additional search query, and (ii) the videos identified in response to submission of the search query.

Assignees

• Google Inc

Inventors

• Shlens Jonathon
• Le Quoc V
• Corrado Gregory Sean
• Ranzato Marc'Aurelio

Classifications

• G06F16/739 Primary

  in form of a video summary, e.g. the video summary being a video sequence, a composite still image or having synthesized frames · CPC title

• G06F18/2413

  based on distances to training or reference patterns · CPC title

• G06F18/28

  Determining representative reference patterns, e.g. by averaging or distorting; Generating dictionaries · CPC title

• G06F18/22

  Matching criteria, e.g. proximity measures · CPC title

• G06F18/214

  Generating training patterns; Bootstrap methods, e.g. bagging or boosting · CPC title