Feature-based video annotation

Patent metadata

Field	Value
Publication number	US-9779304-B2
Application number	US-201514823946-A
Country	US
Kind code	B2
Filing date	Aug 11, 2015
Priority date	Aug 11, 2015
Publication date	Oct 3, 2017
Grant date	Oct 3, 2017

Abstract

A system and methodology provide for annotating videos with entities and associated probabilities of existence of the entities within video frames. A computer-implemented method identifies an entity from a plurality of entities identifying characteristics of video items. The computer-implemented method selects a set of features correlated with the entity based on a value of a feature of a plurality of features, determines a classifier for the entity using the set of features, and determines an aggregation calibration function for the entity based on the set of features. The computer-implemented method selects a video frame from a video item, where the video frame having associated features, and determines a probability of existence of the entity based on the associated features using the classifier and the aggregation calibration function.

First claim

The invention claimed is:

1. A computer-implemented method for annotating videos with entities and associated probabilities of existence of the entities within video frames, the method comprising:
identifying an entity from a plurality of entities identifying characteristics of video items;
from a plurality of features characterizing the video items, selecting a set of features correlated with the entity based on a value of each feature;
determining a classifier for the entity using the set of features, the classifier being a linear fusion model for the entity based on the set of features and determining the classifier comprising determining a weight vector for each feature of the set of features;
determining an aggregation calibration function for the entity based on the set of features, the aggregation calibration function calibrating a fusion score to a probability of the entity being central to an individual video frame;
selecting a video frame from a video item, the video frame having associated features; and
determining a probability of existence of the entity based on the associated features using the classifier and the aggregation calibration function.

2. The method of claim 1 , further comprising labeling the video frame with the entity and the probability of existence of the entity.

3. The method of claim 1 , wherein the step of selecting a set of features correlated with the entity comprises determining a calibration function that calibrates a value of a feature of the plurality of features to an individual probability of existence of the entity.

4. The method of claim 3 , wherein the step of selecting a set of features correlated with the entity comprises determining the individual probability of existence of the entity based on the value of each feature of the plurality of features, and the individual probabilities of existence of the entity for the plurality of features are normalized.

5. The method of claim 4 , wherein the step of selecting a set of features correlated with the entity further comprises selecting the set of features such that a maximum individual probability of existence of the entity corresponding to each selected feature is at least a given threshold value.

6. The method of claim 1 , wherein the step of determining a classifier for the entity comprises maximizing a precision for the linear fusion model and maintaining a recall to be at least a given threshold value.

7. The method of claim 3 , wherein the step of determining a probability of existence of the entity based on the associated features comprises:
using the calibration function to determine an individual probability of existence of the entity based on each feature associated with the entity;
combining the individual probabilities of existence of the entity based on the associated features by using the classifier to determine the fusion score; and
using the aggregation calibration function to calibrate the fusion score to the probability of existence of the entity.

8. The method of claim 1 , wherein the classifier and the aggregation calibration function are determined by a machine learning model, further comprising selecting training data comprising a set of video items, each video item having at least a feature of the set of features, and providing the training data to the machine learning model.

9. The method of claim 1 , further comprising determining a probability of existence for each entity of the plurality of entities based on the associated features for the video frame.

10. The method of claim 9 , further comprising determining the probability of existence for each entity of the plurality of entities based on the associated set of features for each video frame of the video item.

11. The method of claim 1 , further comprising:
receiving a search query from a user; and
providing the video item including the video frame to the user in response to determining the search query matches the entity.

12. A system comprising:
a processor for executing computer program instructions; and
a non-transitory computer-readable storage medium comprising computer program instructions executable by the processor, the computer program instructions comprising:
instructions for identifying an entity from a plurality of entities identifying characteristics of video items;
instructions for, from a plurality of features characterizing the video items, selecting a set of features correlated with the entity based on a value of each feature;
instructions for determining a classifier for the entity using the set of features, the classifier being a linear fusion model for the entity based on the set of features and determining the classifier comprising determining a weight vector for each feature of the set of features;
instructions for determining an aggregation calibration function for the entity based on the set of features, the aggregation calibration function calibrating a fusion score to a probability of the entity being central to an individual video frame;
instructions for selecting a video frame from a video item, the video frame having associated features; and
instructions for determining a probability of existence of the entity based on the associated features using the classifier and the aggregation calibration function.

13. The system of claim 12 , wherein the computer program instructions further comprises instructions for labeling the video frame with the entity and the probability of existence of the entity.

14. The system of claim 12 , wherein the computer program instructions for selecting a set of features correlated with the entity comprises instructions for determining a calibration function that calibrates a value of a feature of the plurality of features to an individual probability of existence of the entity.

15. The system of claim 14 , wherein the computer instructions for selecting a set of features correlated with the entity comprises determining the individual probability of existence of the entity based on the value of each feature of the plurality of features, and the individual probabilities of existence of the entity for the plurality of features are normalized.

16. The system of claim 12 , wherein the computer program instructions further comprises instructions for determining a probability of existence for each entity of the plurality of entities based on the associated features for the video frame.

17. The system of claim 12 , wherein the computer program instructions further comprises:
instructions for receiving a search query from a user; and
instructions for providing the video item including the video frame to the user in response to determining the search query matches the entity.

18. A non-transitory computer-readable storage medium comprising computer program instructions executable by a processor, the computer program instructions comprising:
instructions for identifying an entity from a plurality of entities identifying characteristics of video items;
instructions for, from a plurality of features characterizing the video items, selecting a set of features correlated with the entity based on a value of each feature;
instructions for determining a classifier for the entity using the set of features, the classifier being a linear fusion model for the entity based on the set of features and determining the classifier comprising determining a weight vector for each feature of the set of features;
instructions for determining an aggregation calibration function for the entity based on the set of features, the aggregation ca

Assignees

• Google Inc

Inventors

• Varadarajan Balakrishnan
• Toderici George Dan
• Natsev Apostol
• Khandelwal Nitin
• Vijayanarasimhan Sudheendra
• Yang Weilong
• Shetty Sanketh

Classifications

• G06V20/70

  Labelling scene content, e.g. deriving syntactic or semantic representations · CPC title

• G06V20/41

  Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items (segmenting video sequences G06V20/49) · CPC title

• G06F16/783 Primary

  using metadata automatically derived from the content · CPC title

• G06F18/214

  Generating training patterns; Bootstrap methods, e.g. bagging or boosting · CPC title

• G06F18/2413

  based on distances to training or reference patterns · CPC title