Enriching online videos by content detection, searching, and information aggregation

Patent metadata

Field	Value
Publication number	US-9443147-B2
Application number	US-76711410-A
Country	US
Kind code	B2
Filing date	Apr 26, 2010
Priority date	Apr 26, 2010
Publication date	Sep 13, 2016
Grant date	Sep 13, 2016

Abstract

Many internet users consume content through online videos. For example, users may view movies, television shows, music videos, and/or homemade videos. It may be advantageous to provide additional information to users consuming the online videos. Unfortunately, many current techniques may be unable to provide additional information relevant to the online videos from outside sources. Accordingly, one or more systems and/or techniques for determining a set of additional information relevant to an online video are disclosed herein. In particular, visual, textual, audio, and/or other features may be extracted from an online video (e.g., original content of the online video and/or embedded advertisements). Using the extracted features, additional information (e.g., images, advertisements, etc.) may be determined based upon matching the extracted features with content of a database. The additional information may be presented to a user consuming the online video.

First claim

What is claimed is:

1. A method of identifying digital content relevant to an online video, comprising:
identifying, by one or more processing units, promotional content embedded within the online video, the promotional content being generated by a first content producer;
extracting, by the one or more processing units, a feature from the promotional content embedded in the online video, the feature comprising at least one of a textual feature, a visual feature, or an audio feature;
searching, by the one or more processing units, a database to identify additional information based upon the feature, the additional information being generated by a second content producer different than the first content producer;
identifying, by the one or more processing units, candidates from the additional information, the candidates including at least two of a textual feature, a visual feature, or an audio feature;
aggregating, by the one or more processing units, the candidates into an aggregate video; and
outputting, by the one or more processing units, the aggregate video for presentation.

2. The method of claim 1 , wherein outputting the aggregate video for presentation comprises:
presenting the aggregate video and the promotional content sequentially.

3. The method of claim 1 , the visual feature comprising a color histogram.

4. The method of claim 1 , the extracting the feature comprising at least one of:
extracting the textual feature using an OCR text recognition technique; or
extracting the visual feature using a scale-invariant feature transformation.

5. The method of claim 1 , the searching the database to identify the additional information comprising:
executing a multi modal relevance matching algorithm against the database using the feature to determine the additional information.

6. The method of claim 1 , the aggregate video being formatted to be embedded in the online video.

7. The method of claim 1 , the extracting the feature comprising:
parsing the promotional content embedded in the online video into one or more segments, each of the one or more segments comprising a sequence of frames;
for respective segments, extracting one or more keyframes; and
extracting the feature from the one or more keyframes of respective segments.

8. The method of claim 7 , the extracting one or more keyframes comprising at least one of:
selecting a middle frame as a keyframe; or
selecting a frame having a video quality above a threshold as the keyframe.

9. The method of claim 1 , wherein the aggregate video comprises advertising information associated with the promotional content embedded within the online video.

10. A system for identifying digital content relevant to an online video, comprising:
one or more processing units; and
a memory storing:
a parsing component configured to, when executed by the one or more processing units:
select a frame of an online video as a keyframe based upon the frame having a video quality above a threshold;

a feature extraction component configured to, when executed by the one or more processing units:
identify promotional content embedded within the keyframe, the promotional content being generated by a first content producer; and
extract a feature from at least the promotional content embedded within the keyframe, the feature comprising at least one of a textual feature, a visual feature, or an audio feature;

an information extraction component configured to, when executed by the one or more processing units:
search a database to identify additional information based upon the feature, the additional information being generated by a second content provider different than the first content producer;
identify candidates from the additional information determined to be associated with the feature extracted from the promotional content, the candidates including at least two of a textual feature, a visual feature, or an audio feature; and
in response to identifying the candidates, automatically aggregate the candidates into an aggregate video; and

a presentation component configured to, when executed by the one or more processing units:
present the aggregate video.

11. The system of claim 10 , the presentation component further configured to:
present the online video sequentially with the aggregate video.

12. The system of claim 11 , the aggregate video being different than the promotional content.

13. The system of claim 10 , the parsing component further configured to:
parse the online video into one or more segments, a segment comprising a sequence of frames; and for respective segments, extract one or more keyframes.

14. The system of claim 10 , the information extraction component further configured to:
execute a multi modal relevance matching algorithm against the database using the feature to determine the additional information.

15. The system of claim 14 , the multimodal relevance matching algorithm further configured to:
perform a text based search algorithm upon the database using the textual feature to determine a first list of candidate additional information;
perform a visual feature matching algorithm upon the database using the visual feature to determine a second list of candidate additional information; and
perform a linear combination of the first list and second list to generate a set of additional information.

16. A computer-readable storage device comprising processor-executable instructions that, when executed, cause a computing device to perform a method for identifying digital content relevant to an online video, the method comprising:
selecting a frame of the online video as a keyframe based upon the frame having a video quality above a threshold;
identifying promotional content embedded within the keyframe, the promotional content being generated by a first content producer;
extracting a feature from at least the promotional content embedded in the keyframe, the feature comprising at least one of a textual feature, a visual feature, or an audio feature;
searching a database to identify additional information based upon the feature, the additional information being generated by a second content producer different than the first content producer;
identifying candidates from the additional information determined to be associated with the feature extracted from the promotional content, the candidates including at least two of a texture feature, a visual feature, or an audio feature;
in response to identifying the candidates, automatically aggregating the candidates into an aggregate video; and
outputting the aggregate video for presentation.

17. The computer-readable storage device of claim 16 , the method comprising:
presenting the online video sequentially with the aggregate video.

Assignees

• Mei Tao
• Hua Xian-Sheng
• Li Shipeng
• Microsoft Technology Licensing Llc

Inventors

• Mei Tao
• Hua Xian-Sheng
• Li Shipeng

Classifications

• H04N21/4622

  Retrieving content or additional data from different sources, e.g. from a broadcast channel and the Internet (web site content organization and management for information retrieval from the Internet G06F16/958; transmission by internet of broadcast information H04H60/82; stock exchange data over packet-switching network H04L12/1804; push services including data channel over packet-switching network H04L12/1859) · CPC title

• G06F17/30017

  Physics · mapped topic

• G06K9/00765 Primary

  Physics · mapped topic

• G06Q30/02 Primary

  Marketing; Price estimation or determination; Fundraising · CPC title

• G06K9/00751

  Physics · mapped topic