Video frame annotation

Patent metadata

Field	Value
Publication number	US-9627004-B1
Application number	US-201514883461-A
Country	US
Kind code	B1
Filing date	Oct 14, 2015
Priority date	Oct 14, 2015
Publication date	Apr 18, 2017
Grant date	Apr 18, 2017

Abstract

A system and methodology provide for annotating videos with entities and associated probabilities of existence of the entities within video frames. A computer-implemented method selects an entity from a plurality of entities identifying characteristics of a video item, where the video item has associated metadata. The computer-implemented method receives probabilities of existence of the entity in video frames of the video item, and selects a video frame determined to comprise the entity responsive to determining the video frame having a probability of existence of the entity greater than zero. The computer-implemented method determines a scaling factor for the probability of existence of the entity using the metadata of the video item, and determines an adjusted probability of existence of the entity by using the scaling factor to adjust the probability of existence of the entity. The computer-implemented method labels the video frame with the adjusted probability of existence.

First claim

The invention claimed is:

1. A computer-implemented method for annotating videos with entities and associated probabilities of existence of the entities within video frames, the method comprising:
selecting an entity from a plurality of entities identifying characteristics of a video item, the video item having associated metadata;
receiving probabilities of existence of the entity in video frames of the video item;
selecting a video frame determined to comprise the entity responsive to determining the video frame having a probability of existence of the entity greater than zero;
determining a scaling factor for the probability of existence of the entity using the metadata of the video item;
determining an adjusted probability of existence of the entity by using the scaling factor to adjust the probability of existence of the entity; and
labeling the video frame with the adjusted probability of existence.

2. The method of claim 1 , wherein the metadata comprises a centrality of the entity indicating an importance of the entity, and the step of determining a scaling factor comprises:
identifying a maximum probability of existence of the entity in the video; and
calculating a ratio of the centrality of the entity to the maximum probability of existence of the entity in the video as the scaling factor.

3. The method of claim 2 , further comprising:
multiplying the probability of existence by the scaling factor to determine an interim probability of existence;
comparing the interim probability of existence to the probability of existence, and;
determining the adjusted probability of existence as a greater between the interim probability of existence and the probability of existence.

4. The method of claim 1 , wherein the scaling factor is based on a linear fusion model, further comprising measuring a weight vector for each metadata, the weight vector representing an influence of the metadata on the probability of existence of the entity.

5. The method of claim 4 , further comprising determining a classifier comprising a set of weight vectors, the set of weight vectors including a first weight vector for retention statistics for the video item, a weight vector for video-level features of the entity for the video item, and a weight vector for frame-level features of the entity for the video item.

6. The method of claim 1 , wherein the scaling factor is based on a classifier determined by a machine learning model, the method further comprising providing training data comprising a set of video items, metadata associated with the set of video items, and probabilities of existence of items associated with the set of video items to the machine learning model.

7. The method of claim 1 , further comprising:
receiving a search query from a user; and
providing the video item including the video frame to the user in response to determining that the search query matches the entity.

8. A system comprising:
a processor for executing computer program instructions; and
a non-transitory computer-readable storage medium comprising computer program instructions executable by the processor, the computer program instructions comprising:
selecting an entity from a plurality of entities identifying characteristics of a video item, the video item having associated metadata;
receiving probabilities of existence of the entity in video frames of the video item;
selecting a video frame determined to comprise the entity responsive to determining the video frame having a probability of existence of the entity greater than zero;
determining a scaling factor for the probability of existence of the entity using the metadata of the video item;
determining an adjusted probability of existence of the entity by using the scaling factor to adjust the probability of existence of the entity; and
labeling the video frame with the adjusted probability of existence.

9. The system of claim 8 , wherein the metadata comprises a centrality of the entity indicating an importance of the entity, and the step of determining a scaling factor comprises:
identifying a maximum probability of existence of the entity in the video; and
calculating a ratio of the centrality of the entity to the maximum probability of existence of the entity in the video as the scaling factor.

10. The system of claim 9 , wherein the computer program instructions further comprises:
multiplying the probability of existence by the scaling factor to determine an interim probability of existence;
comparing the interim probability of existence to the probability of existence, and;
determining the adjusted probability of existence as a greater between the interim probability of existence and the probability of existence.

11. The system of claim 8 , wherein the scaling factor is based on a linear fusion model, and wherein the computer program instructions further comprises measuring a weight vector for each metadata, the weight vector representing an influence of the metadata on the probability of existence of the entity.

12. The system of claim 11 , wherein the computer program instructions further comprises determining a classifier comprising a set of weight vectors, the set of weight vectors including a first weight vector for retention statistics for the video item, a weight vector for video-level features of the entity for the video item, and a weight vector for frame-level features of the entity for the video item.

13. The system of claim 8 , wherein the scaling factor is based on a classifier determined by a machine learning model, and wherein the computer program instructions further comprises providing training data comprising a set of video items, metadata associated with the set of video items, and probabilities of existence of items associated with the set of video items to the machine learning model.

14. The system of claim 8 , wherein the computer program instructions further comprises:
receiving a search query from a user; and
providing the video item including the video frame to the user in response to determining that the search query matches the entity.

15. A non-transitory computer-readable storage medium comprising computer program instructions executable by a processor, the computer program instructions comprising:
selecting an entity from a plurality of entities identifying characteristics of a video item, the video item having associated metadata;
receiving probabilities of existence of the entity in video frames of the video item;
selecting a video frame determined to comprise the entity responsive to determining the video frame having a probability of existence of the entity greater than zero;
determining a scaling factor for the probability of existence of the entity using the metadata of the video item;
determining an adjusted probability of existence of the entity by using the scaling factor to adjust the probability of existence of the entity; and
labeling the video frame with the adjusted probability of existence.

16. The non-transitory computer-readable storage medium of claim 15 , wherein the metadata comprises a centrality of the entity indicating an importance of the entity, and the step of determining a scaling factor comprises:
identifying a maximum probability of existence of the entity in the video; and
calculating a ratio of the centrality of the entity to the maximum probability of existence of the entity in the video as the scaling factor.

17. The non-transitory computer-readable storage medium of claim 16 , wherein the computer program instructions further comprises:
multi

Assignees

• Google Inc

Inventors

• Varadarajan Balakrishnan
• Shetty Sanketh
• Natsev Apostol
• Khandelwal Nitin
• Yang Weilong
• Vijayanarasimhan Sudheendra
• Gu Weihsin
• Muscettola Nicola

Classifications

• G06V10/7753

  Incorporation of unlabelled data, e.g. multiple instance learning [MIL] · CPC title

• G06V10/764

  using classification, e.g. of video objects · CPC title

• G06V20/46 Primary

  Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames · CPC title

• G06F18/2155

  characterised by the incorporation of unlabelled data, e.g. multiple instance learning [MIL], semi-supervised techniques using expectation-maximisation [EM] or naïve labelling · CPC title

• G06F18/2415

  based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate · CPC title