Computerized machine learning of interesting video sections

Patent metadata

Field	Value
Publication number	US-9646227-B2
Application number	US-201414445463-A
Country	US
Kind code	B2
Filing date	Jul 29, 2014
Priority date	Jul 29, 2014
Publication date	May 9, 2017
Grant date	May 9, 2017

Abstract

This disclosure describes techniques for training models from video data and applying the learned models to identify desirable video data. Video data may be labeled to indicate a semantic category and/or a score indicative of desirability. The video data may be processed to extract low and high level features. A classifier and a scoring model may be trained based on the extracted features. The classifier may estimate a probability that the video data belongs to at least one of the categories in a set of semantic categories. The scoring model may determine a desirability score for the video data. New video data may be processed to extract low and high level features, and feature values may be determined based on the extracted features. The learned classifier and scoring model may be applied to the feature values to determine a desirability score associated with the new video data.

First claim

What is claimed is:

1. A method comprising:
receiving, by one or more computing devices, video data;
extracting, by at least one of the one or more computing devices, a plurality of features from the video data;
determining, by at least one of the one or more computing devices, a first set of feature values associated with the plurality of features, the first set of feature values for training a classifier and a scoring model;
determining, by at least one of the one or more computing devices, a second set of feature values based on applying the classifier to the video data;
training, by at least one of the one or more computing devices, the scoring model based on the first set of feature values and the second set of feature values;
using the scoring model to determine a plurality of desirability scores associated with the video data, wherein an individual desirability score indicative of video quality is associated with an individual video frame in the video data;
identifying video frames in the video data that have a desirability score above a predetermined threshold desirability score;
analyzing the video data to determine, in association with the video frames, changes in camera motion and changes in object motion; and
locating, based at least in part on the changes in camera motion and the changes in object motion, boundaries in the video data to produce one or more video segments, wherein:
an individual video segment includes at least one video frame that has the desirability score above the predetermined threshold desirability score, and
the locating the boundaries in the video data comprises determining that object motion intensity of a first video frame and object motion intensity of a second video frame differ by a predetermined threshold.

2. The method of claim 1 , wherein:
the plurality of features includes low level features and high level features;
the first set of feature values includes at least a low level feature value and a first high level feature value; and
the second set of feature values includes at least one second high level feature value that is not included in the first set of feature values.

3. The method of claim 2 , wherein the first set of feature values represents a plurality of derivative feature values, wherein individual derivative feature values of the plurality of derivative feature values are derived from at least some low level feature values or high level feature values.

4. The method of claim 1 , wherein the second set of feature values represents probabilities that the video data belongs to at least one semantic category of a predefined set of semantic categories.

5. The method of claim 1 , wherein the video data comprises a video collection.

6. The method of claim 1 , wherein the first set of feature values represents at least one of video collection level feature values, video file level feature values, video segment level feature values, or video frame level feature values and the second set of feature values represents video file level feature values.

7. A system comprising:
memory;
one or more processors; and
one or more modules stored in the memory and executable by the one or more processors, the one or more modules including:
an extracting module configured to extract features from video data and determine a first set of feature values based on the extracted features;
a classifying module configured to apply a classifier to the first set of feature values to determine a second set of feature values and to use the second set of feature values to determine a probability that the video data belongs to at least one semantic category of a predefined set of semantic categories;
a scoring module configured to apply a scoring model, based at least in part on the at least one semantic category, to the first set of feature values and the second set of feature values to determine a plurality of desirability scores for the video data, wherein an individual desirability score indicative of video quality is associated with an individual video frame in the video data; and
a segmenting module configured to:
identify video frames in the video data that have a desirability score above a predetermined threshold desirability score;
analyze the video data to determine, in association with the video frames, changes in camera motion and changes in object motion; and
locate, based at least in part on the changes in camera motion and the changes in object motion, boundaries in the video data to produce one or more video segments, wherein:
an individual video segment includes at least one video frame that has the desirability score above the predetermined threshold desirability score, and
the locating the boundaries in the video data comprises determining that object motion intensity of a first video frame and object motion intensity of a second video frame differ by a predetermined threshold.

8. The system of claim 7 , wherein the features include at least one of:
exposure quality;
saturation quality;
hue variety;
stability;
face detection;
face recognition;
face tracking;
saliency analysis;
audio power analysis;
speech detection; or
motion analysis.

9. The system of claim 7 , wherein the one or more modules further include a post-processing module configured to:
rank the one or more video segments based at least in part on desirability scores of video frames included in an individual video segment; and
create a highlight video based at least in part on the ranking.

10. One or more computer-readable storage media encoded with instructions that, when executed by a processor, perform acts comprising:
receiving video data including a plurality of video frames;
extracting a plurality of features from individual video frames of the plurality of video frames to determine a first set of feature values associated with the individual video frames;
applying a classifier to the first set of feature values to determine a second set of feature values associated with the individual video frames;
using the second set of feature values to determine individual probabilities that the individual video frames belong to at least one semantic category of a predefined set of semantic categories;
applying a scoring model to the first set of feature values and the second set of feature values to determine desirability scores associated with the individual video frames;
identifying a subset of the plurality of video frames in the video data that have a desirability score above a predetermined threshold desirability score;
analyzing the video data to determine, in association with the subset of video frames, changes in camera motion and changes in object motion; and
locating, based at least in part on the changes in camera motion and the changes in object motion, boundaries in the video data to produce one or more video segments, wherein:
an individual video segment includes at least one video frame that has the desirability score above the predetermined threshold desirability score, and
the locating the boundaries in the video data comprises determining that object motion intensity of a first video frame and object motion intensity of a second video frame differ by a predetermined threshold.

11. The computer-readable storage media of claim 10 , wherein the first set of feature values represents feature values associated with low level features, high level features, and derivatives of the low level features and high level features.

12. The computer-readable storage media of claim 10 , wherein the video data comprises video files and the acts further comprise determining a desirability score for individual v

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Suri Nitin
• Hua Xian-Sheng
• Wang Tzong-Jhy
• Sproule William D
• Ivory Andrew S
• Li Jin

Classifications

• G06V10/774

  Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting · CPC title

• G06F18/214

  Generating training patterns; Bootstrap methods, e.g. bagging or boosting · CPC title

• G06T7/20 Primary

  Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51) · CPC title

• G06T2207/10016

  Video; Image sequence · CPC title

• G06K9/00744

  Physics · mapped topic