Classification using concept ranking according to negative exemplars

Patent metadata

Field	Value
Publication number	US-9619521-B1
Application number	US-201314142976-A
Country	US
Kind code	B1
Filing date	Dec 30, 2013
Priority date	Dec 30, 2013
Publication date	Apr 11, 2017
Grant date	Apr 11, 2017

Abstract

A segmentation annotation technique for media items is disclosed herein. Given a weakly labeled media item, spatiotemporal masks may be generated for each of the concepts with which it is labeled. Segments may be ranked by the likelihood that they correspond to a given concept. The ranked concept segments may be utilized to train a classifier that, in turn, may be used to classify untagged or new media items.

First claim

The invention claimed is:

1. A system comprising:
a database for storing a plurality of media items;
one or more computers connected to the database and configured to:
obtain the plurality of media items, each of the plurality of media items being identified as either (i) a concept media item that has been classified as a media item in which a particular visual concept appears or (ii) a non-concept media item that has been classified as a media item in which the particular visual concept does not appear;
obtain a plurality of concept segments, wherein each of the concept segments is a segment that has been extracted from a concept media item;
obtain a plurality of non-concept segments, wherein each non-concept segment is a segment that has been extracted from a non-concept media item, wherein each concept segment and each non-concept segment is represented in a feature space;
for each non-concept segment, identify a closest concept segment, wherein the closest concept segment is the concept segment that is closest to the non-concept segment of any of the plurality of concept segments, wherein the closest concept segment is identified based upon pairwise distances between all of the concept segments and all of the non-concept segments in the feature space;
determine, for each concept segment, a respective count of instances in which the concept segment is identified as the closest concept segment to one of the non-concept segments;
rank each concept segment such that the ranking reflects a respective likelihood that the concept segment contains the particular visual concept by ranking the concept segments such that concept segments having lower counts are favored over concept segments having higher counts; and
label concept segments that are below a threshold rank in the ranking as not containing the particular visual concept.

2. The system of claim 1 , the one or more computers further configured to:
train a classifier based on at least a portion of the ranked concept segments and the non-concept segments; and
classify a new media item using the classifier.

3. The system of claim 2 , wherein the classifier is trained based on all of the ranked concept segments, and wherein the concept segments that are below the threshold rank are used as non-concept segments to train the classifier.

4. The system of claim 1 , wherein each of the plurality of media items is identified as either a concept media item or a non-concept media item based on a weak label assigned to the media item before the obtaining of the media item.

5. The system of claim 1 , the one or more computers further configured to segment the plurality of media items to generate the concept and non-concept segments.

6. The system of claim 1 , wherein:
each of the media items in the plurality of media items is a video; and
each segment obtained for each of the videos is a spatiotemporal (3D) volume that is represented as a point in a high-dimensional feature space using a set of standard features computed over the segment.

7. A computer-implemented method, comprising:
obtaining, by one or more computers, a plurality of media items, each of the plurality of media items being identified as either (i) a concept media item that has been classified as a media item in which a particular visual concept appears or (ii) a non-concept media item that has been classified as a media item in which the particular visual concept does not appear;
obtaining, by the one or more computers, a plurality of concept segments, wherein each of the concept segments is a segment that has been extracted from a concept media item;
obtain a plurality of non-concept segments, wherein each non-concept segment is a segment that has been extracted from a non-concept media item, wherein each concept segment and each non-concept segment is represented in a feature space;
for each non-concept segment, identify a closest concept segment, wherein the closest concept segment is the concept segment that is closest to the non-concept segment of any of the plurality of concept segments, wherein the closest concept segment is identified based upon pairwise distances between all of the concept segments and all of the non-concept segments in the feature space;
determine, by the one or more computers and for each concept segment, a respective count of instances in which the concept segment is identified as the closest concept segment to one of the non-concept segments;
ranking, by the one or more computers, each concept segment such that the ranking reflects a respective likelihood that the concept segment contains the particular visual concept by ranking the concept segments such that concept segments having lower counts are favored over concept segments having higher counts; and
labeling, by the one or more computers, concept segments that are below a threshold rank in the ranking as not containing the particular visual concept.

8. The method of claim 7 , further comprising:
training a classifier based on at least a portion of the ranked concept segments and the non-concept segments; and
classifying a new media item using the classifier.

9. The method of claim 8 , wherein the classifier is trained based on all of the ranked concept segments, and wherein the concept segments below the threshold rank are used as non-concept segments to train the classifier.

10. The method of claim 7 , wherein each of the plurality of media items is identified as either a concept media item or a non-concept media item based on a weak label assigned to the media item before the media item was obtained.

11. The method of claim 7 , further comprising segmenting the plurality of media items to generate the concept and non-concept segments.

12. The method of claim 7 , wherein:
each of the media items in the plurality of media items is a video; and
each segment obtained for each of the videos is a spatiotemporal (3D) volume that is represented as a point in a high-dimensional feature space using a set of standard features computed over the segment.

13. One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:
obtaining, by one or more computers, a plurality of media items, each of the plurality of media items being identified as either (i) a concept media item that has been classified as a media item in which a particular visual concept appears or (ii) a non-concept media item that has been classified as a media item in which the particular visual concept does not appear;
obtaining, by the one or more computers, a plurality of concept segments, wherein each of the concept segments is a segment that has been extracted from a concept media item;
obtain a plurality of non-concept segments, wherein each non-concept segment is a segment that has been extracted from a non-concept media item, wherein each concept segment and each non-concept segment is represented in a feature space;
for each non-concept segment, identify a closest concept segment, wherein the closest concept segment is the concept segment that is closest to the non-concept segment of any of the plurality of concept segments, wherein the closest concept segment is identified based upon pairwise distances between all of the concept segments and all of the non-concept segments in the feature space;
determine, by the one or more computers and for each concept segment, a respective count of instances in which the concept segment is identified as the closest concept segment to one of the non-concept segments;
ranking, by the one or more computers, each concept seg

Assignees

• Google Inc

Inventors

• Sukthankar Rahul
• Yagnik Jay

Classifications

• G06N99/005

  Physics · mapped topic

• G06F17/3053 Primary

  Physics · mapped topic

• G06N20/00 Primary

  Machine learning · CPC title

• G06F16/75 Primary

  Clustering; Classification · CPC title

• G06F16/7837

  using objects detected or recognised in the video content · CPC title