TV content segmentation, categorization and identification and time-aligned applications

Patent metadata

Field	Value
Publication number	US-9510044-B1
Application number	US-201113327359-A
Country	US
Kind code	B1
Filing date	Dec 15, 2011
Priority date	Jun 18, 2008
Publication date	Nov 29, 2016
Grant date	Nov 29, 2016

Abstract

Content segmentation, categorization and identification methods are described. Content tracking approaches are illustrated that are suitable for large scale deployment. Time-aligned applications such as multi-language selection, customized advertisements, second screen services and content monitoring applications can be economically deployed at large scales. A client performs fingerprinting, scene change detection, audio turn detection, and logo detection on incoming video and gathers database search results, logos and text to identify and segment video streams into content, promos, and commercials. A learning engine is configured to learn rules for optimal identification and segmentation at each client for each channel and program. Content sensed at the client site is tracked with reduced computation and applications are executed with timing precision. A user interface for time-aligned publishing of content and subsequent usage and interaction on one or more displays is also described.

First claim

What is claimed is:

1. A method for time aligned identification of video content playing on an electronic client device, the method comprising:
receiving by the electronic client device, a video stream for presentation by the electronic client device, the video stream containing a first video content followed by a second video content;
generating by the electronic client device, a first set of signatures for the first video content;
identifying by the electronic client device, the first video content based on an analysis of the first set of signatures compared to reference signatures stored in a database;
detecting by the electronic client device, a scale ratio mapping between video frames of the identified first video content playing on the electronic client device and reference frames stored in the database by calculating a scale ratio along an x axis (Sx) and a difference scale ratio along a y axis (Sy) for a query signature in the first set of signatures and a reliable matching reference signature in the database, wherein the calculated scale ratios Sx and Sy indicate that the video frames of the identified first video content and the reference frames are not in alignment;
responsive to calculating the scale ratios, using the detected scale ratio mapping to generate subsequent signatures for subsequent incoming video frames; and
monitoring the subsequent incoming video frames for the second video content by the electronic client device to determine that the video stream has made a scene change from the first video content to the second video content.

2. The method of claim 1 further comprising:
replacing the second video content beginning at a time the scene change is verified with a third video that is different than the second video content.

3. The method of claim 2 , wherein the second video content is an expected advertisement within an expected content or program stream.

4. The method of claim 1 further comprising:
displaying synchronous information on a selected device.

5. The method of claim 1 further comprising:
providing a user selection of display devices for synchronous display of data in place of the identified second video content;
providing a user choice of content for synchronous stream content in place of the identified second video content; and
providing a user choice of scrolling in time to select content choices including synchronous content and the identified second video content on the client device.

6. The method of claim 1 , wherein identifying a time alignment comprises:
sequence matching and correlation between reference and query fingerprints.

7. The method of claim 1 , wherein identifying a change in multimedia content comprises:
detecting a scene change.

8. The method of claim 1 , wherein verifying a sample of multimedia content comprises:
matching the sample of multimedia content with a first frame or an opening sequence of an expected multimedia content.

9. The method of claim 1 , wherein audio content is corrected for changes in pitch and changes in frequency response.

10. The method of claim 1 , wherein Sx is equal to (QA(x)−QB(x))/(RA(x)−RB(x)) and wherein QA(x)−QB(x) is a difference in query x coordinates of signature pairs A and B and RA(x)−RB(x) is a difference in reference x coordinates of the signature pairs A and B.

11. A method of providing time-aligned services on broadcast non-recorded content, the method comprising:
analyzing by an electronic client device, incoming video of a broadcast live event based on comparisons with opening sequences of programs stored on the electronic client device to determine program details associated with the broadcast live event;
generating by the electronic client device, fingerprints of selected frames of the incoming video;
searching a reference database using the generated fingerprints to identify a likely match for content of the incoming video with content of an original video stored in the reference database; and
correcting by the electronic client device, frame alignment in the incoming video based on a scale ratio mapping by utilizing a scale ratio for an x axis (Sx) and a different scale ratio for a y axis (Sy) between the selected frames of the incoming video and corresponding frames of a matched original video selected from the reference database and for generating subsequent fingerprints based on the scale ratio mapping to enable a time aligned service to overlay on aligned video frames for display on a display of the electronic client device information personalized for the electronic client device.

12. The method of claim 11 further comprises:
searching on the electronic client device a subset reference database that is a subset of a remote database using the generated subsequent fingerprints in queries to minimize computational effort and improve accuracy in identification of the incoming video; and
searching on the remote database in response to a match not being found in the subset reference database.

13. The method of claim 11 further comprises:
using search reports and content analytics to determine a classification of content at time nodes and a similarity between the time nodes generated for the incoming video;
generating edge weights between the time nodes, wherein the edge weights represent the similarity between two edge connected time nodes; and
performing graph based segmentation to segment video based on the classification and similarity between the time nodes.

14. The method of claim 11 further comprises:
determining a geometric correlation of video signatures and associated data comprising geometric coordinates and scale information between the incoming video and the matched original video to represent transformation of the matched original video that approximate the incoming video.

15. The method of claim 11 , wherein video segments of the incoming video represent chapters of a program, advertisements, and promotions.

16. The method of claim 11 further comprises:
generating the fingerprints of incoming video including a determined region's location and scale information;
determining a reliable match of content from incoming video with the reference database; and
generating frame alignment information that includes pixel offsets along the x axis and along the y axis, the scale ratio for the x axis (Sx), and the scale ratio for the y axis (Sy) to represent transformation of the matched original video that approximate the incoming video.

17. The method of claim 16 , wherein generating frame alignment information comprises:
generating pixel locations for each fingerprint generated for the incoming video;
calculating the scale ratios Sx and Sy based on pixel differences between a query signature pair and a reference signature pair along each axis;
summing up the scale ratios calculated on each axis and averaging while excluding outliers; and
calculating a pixel offset on each axis for matching pairs of signatures based on the scale ratio on that axis and the determined region's location for the query signature pair and the reference signature pair.

18. A method to identify and track video content by an electronic user device, the method comprising:
detecting by the electronic user device, a scale ratio mapping between incoming video frames received by the electronic user device and reference video frames stored in a reference database by calculating a scale ratio along an x axis (Sx) and a different scale ratio along a y axis (Sy) for a query signature in an incoming video frame and a reliable matching

Assignees

• Pereira Jose Pio
• Kulkarni Sunil Suresh
• Bolgarov Oleksiy
• Ramanathan Prashant
• Merchant Shashank
• Stojancic Mihailo
• Gracenote Inc

Inventors

• Pereira Jose Pio
• Kulkarni Sunil Suresh
• Bolgarov Oleksiy
• Ramanathan Prashant
• Merchant Shashank
• Stojancic Mihailo

Classifications

• H04N21/8352

  involving content or source identification data, e.g. Unique Material Identifier [UMID] · CPC title

• H04N21/44008 Primary

  involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream (arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59) · CPC title

• H04H60/37

  for identifying segments of broadcast information, e.g. scenes or extracting programme ID · CPC title

• H04H60/59

  of video {(recognising characters or patterns in general G06F18/00, G06V20/00)} · CPC title

• H04H60/56

  Arrangements characterised by components specially adapted for monitoring, identification or recognition covered by groups H04H60/29-H04H60/54 · CPC title