Methods and systems for aggregation and organization of multimedia data acquired from a plurality of sources

Patent metadata

Field	Value
Publication number	US-2016227282-A1
Application number	US-201615095718-A
Country	US
Kind code	A1
Filing date	Apr 11, 2016
Priority date	Oct 10, 2013
Publication date	Aug 4, 2016
Grant date	—

Abstract

A system for aggregation and organization of multimedia data acquired from a plurality of sources can include a data collection element, a multimedia segmentation element, a memory, an extraction element, and a user interface. The data collection element can be configured to receive multimedia data. The multimedia segmentation element can be coupled to the data collection element and can be configured to separate the multimedia data into a plurality of multimedia data segments. The memory can be configured to store hierarchical category information having a plurality of categories. The extraction element can be coupled to the multimedia segmentation element and the memory and can be configured to associate at least one of the plurality of categories with each of the plurality of multimedia data segments.

First claim

1 . A system for aggregation and organization of multimedia data acquired from a plurality of sources, comprising:
a data collection element configured to receive multimedia data; a multimedia segmentation element, coupled to the data collection element, and configured to separate the multimedia data into a plurality of multimedia data segments; a memory configured to store hierarchical category information having a plurality of categories; an extraction element, coupled to the multimedia segmentation element and the memory, configured to associate at least one of the categories with each of the multimedia data segments; and a user interface, coupled to the extraction element, configured to provide a user with at least one of the multimedia data segments associated with a corresponding one of the categories in the hierarchical category information.

2 . The system of claim 1 , wherein the data collection element further comprises a recorder configured to record audiovisual data.

3 . The system of claim 2 , wherein the recorder is configured to record at least one of analog video content and digital video content.

4 . The system of claim 2 , wherein the recorder further comprises a scheduling routine configured to query an electronic program guide feed and schedule a recording job for a news program.

5 . The system of claim 1 , wherein the data collection element further comprises a web crawler.

6 . The system of claim 1 , wherein the multimedia segmentation element further comprises a closed caption reader configured to identify multimedia data boundaries.

7 . The system of claim 6 , wherein the multimedia segmentation element further comprises a dynamic time warping algorithm configured to align audiovisual data and closed caption data.

8 . The system of claim 1 , wherein the multimedia segmentation element further comprises a speech recognition element.

9 . The system of claim 1 , wherein the multimedia segmentation element further comprises a shot detection element configured to identify shot boundaries.

10 . The system of claim 1 , wherein at least one of the plurality of categories comprises a long-term news event.

11 . The system of claim 1 , wherein at least one of the plurality of categories comprises a breaking news event.

12 . The system of claim 1 , wherein the extraction element further comprises a news video frame layout identifier.

13 . The system of claim 1 , wherein the extraction element further comprises a visual matching element.

14 . The system of claim 1 , wherein the extraction element further comprises a text-matching element.

15 . The system of claim 14 , wherein the text-matching algorithm further comprises a term-frequency and inverse-document-frequency (TF-IDF) algorithm.

16 . The system of claim 1 , wherein the extraction element further comprises a video character recognition element configured to recognize text superimposed on video frames.

17 . The system of claim 1 , wherein the extraction element further comprises a speaker identification element.

18 . The system of claim 1 , wherein the user interface further comprises a text entry box.

19 . The system of claim 1 , wherein the user interface further comprises a cross-domain bookmarklet feature.

20 . The system of claim 1 , wherein the user interface further comprises a filter feature.

21 . The system of claim 1 , wherein the user interface is further configured utilizing a physical simulation model to provide the user with at least one of the multimedia data segments based on an interaction between two or more categories associated with the provided multimedia data segments.

22 . The system of claim 21 , wherein the categories can include a topic, a named speaker, or other information extracted from the multimedia data segment.

23 . A method of aggregation and organization of multimedia data acquired from a plurality of sources, comprising:
receiving and separating the multimedia data into a plurality of multimedia data segments; storing hierarchical category information having a plurality of categories; associating at least one of the categories with each of the multimedia data segments; and providing a user with at least one of the multimedia data segments associated with corresponding categories in the hierarchical category information.

24 . The method of claim 23 , wherein receiving multimedia data further comprises recording audiovisual data.

25 . The method of claim 23 , wherein receiving multimedia data further comprises crawling web sites to acquire the multimedia data.

26 . The method of claim 23 , wherein segmenting the multimedia data further comprises identifying multimedia data boundaries.

27 . The method of claim 26 , wherein identifying multimedia data boundaries further comprises reading closed captions.

28 . The method of claim 26 , wherein identifying multimedia data boundaries further comprises applying a dynamic time warping algorithm.

29 . The method of claim 23 , wherein segmenting the multimedia data further comprises identifying shot boundaries.

30 . The method of claim 23 , wherein associating at least one of the categories with each of the multimedia data segments further comprises visually matching at least a portion of the multimedia data and/or textually matching a least a portion of the multimedia data.

31 . The method of claim 23 , further comprising extracting category names and speaker names by extracting textual information from superimposed text on video image frames and associating speaker names with at least one of a plurality of faces and a plurality of speech segments having at least one of high visual similarity and speech similarity.

32 . The method of claim 23 , further compromising linking at least one of a plurality of multimedia segments and a plurality of segment clusters to relevant trends in social media.

33 . The system of claim 1 , wherein the extraction element is further configured to detect speaker names and specific times when a named speaker speaks during a multimedia data segment.

34 . The system of claim 1 , wherein the extraction element further compromises an element configured to link at least one of a plurality of multimedia segments and a plurality of segment clusters to relevant trends in social media.

Assignees

• Univ Columbia

Inventors

• Chang Shih-Fu
• Jou Brendan
• Li Hongzhi
• Ellis Joseph G
• Morozoff-Abezgauz Daniel

Classifications

• H04N21/4334

  Recording operations (recording of a television signal H04N5/76; arrangements for recording or accumulating broadcast information or broadcast-related information H04H60/27) · CPC title

• H04N21/858

  Linking data to content, e.g. by linking an URL to a video object, by creating a hotspot · CPC title

• H04N21/4884

  for displaying subtitles · CPC title

• H04N21/4332

  by placing content in organized collections, e.g. local EPG data repository (interfaces, Database management systems or updating for information retrieval G06F16/23; details of retrieval of video data and associated meta data in video database G06F16/739) · CPC title

• H04N21/4821

  using a grid, e.g. sorted out by channel and broadcast time · CPC title