System and method for video program recognition

Patent metadata

Field	Value
Publication number	US-9432702-B2
Application number	US-201414325004-A
Country	US
Kind code	B2
Filing date	Jul 7, 2014
Priority date	Jul 7, 2014
Publication date	Aug 30, 2016
Grant date	Aug 30, 2016

Abstract

A method for video program recognition includes recording video sequences of each of a plurality of channels and obtaining a set of features of an image captured by a user device. The method also includes performing a camera registration operation on the video sequences to select video frames representing the recorded video sequences based on key video frames from a registered cameras associated with the recorded video sequences. Further, the method includes extracting features of at least some of the selected plurality of video frames of each channel, storing the extracted features, and comparing the obtained set of features with the stored features of each channel to determine whether there is a match between the obtained set of features of the image and features of a video frame from a specific channel and, when there is a match, determining the specific channel as a recognized channel number.

First claim

What is claimed is:

1. A method for video program recognition on a server system, comprising:
recording, by the server system, video sequences of each of a plurality of channels, wherein the video sequences of each of the plurality of channels are formed by interleaved segments of camera takes filmed with a plurality of original camera setups, and a camera take is a series of frames filmed with one camera setup;
performing, by the server system, a camera registration operation on the video sequences of each of the plurality of channels to select a plurality of video frames representing the recorded video sequences of each of the plurality of channels based on key video frames from a plurality of registered cameras associated with the recorded video sequences;
extracting features of at least some of the selected plurality of video frames of each of the plurality of channels;
storing the extracted features of each of the plurality of channels;
obtaining a set of features of an image captured by a first user device of a video frame from one of the plurality of channels and played on a second user device different from the first user device;
comparing the obtained set of features of the image with the stored features of each of the plurality of channels using an image classifier to determine whether there is a match between the obtained set of features of the image and features of a video frame from a specific channel; and
when there is a match between the obtained set of features of the image and features of a video frame from a specific channel, determining a channel number of the specific channel as a recognized channel number;
wherein performing the camera registration operation further includes:
detecting camera takes in the recorded video sequences of each of the plurality of channels such that the recorded video sequences are separated into a number of sequences with each corresponding to a registered camera that is aligned to one of the plurality of the original camera setups;
forming a registered camera pool of camera takes for the detected camera takes for each of the plurality of channels;
calculating frame difference between consecutive frames using one of a plurality of algorithms;
detecting a new video shot when the frame difference is above a preset threshold value;
selecting a key frame to represent visual content of each detected video shot;
matching the key frame with a last shot of each registered camera based on a matching criterion, wherein when a registered camera is initially detected, a first shot is used as the last shot of the registered camera; and
when the matching criterion is satisfied, adding the key frame to an end of the matching camera take.

2. The method according to claim 1 , wherein:
the first user device is a mobile device and the mobile device captures the image by a camera.

3. The method according to claim 2 , wherein:
the second user device is a TV; and
the TV and the mobile device are associated with one or more multi-screen applications.

4. The method according to claim 2 , wherein:
the mobile device extracts the set of features of the image in same manner as the features of the selected plurality of video frames of each of the plurality of channels are extracted.

5. The method according to claim 1 , further including:
performing an adaptive flow control operation to adjust a throughput of selected plurality of frames from the camera registration operation to match a throughput of extracting features of each of the plurality of channels.

6. The method according to claim 5 , wherein performing the adaptive flow control operation further includes:
calculating a ranking score of a video frame of a channel outputted from the camera registration operation based on at least according to a priority of selection of the channel based on user inquiry and a similarity score between the video frame and a last selected frame of a corresponding camera take; and
during the adaptive flow control operation, selecting a video frame with a highest ranking score for feature extraction.

7. The method according to claim 5 , wherein:
the extracted features of the plurality of video frames includes global features describing an entire frame image and local features using descriptors of local image neighborhoods computed at multiple interest points.

8. The method according to claim 7 , wherein comparing the obtained set of features of the image with the stored features of each of the plurality of channels further includes:
under a processing congestion condition, comparing the obtained set of features of the image with the stored features of each of the plurality of channels using the global features; and
under a processing non-congestion condition, comparing the obtained set of features of the image with the stored features of each of the plurality of channels using the local features.

9. The method according to claim 1 , wherein performing the camera registration operation further includes:
when no matched image is added to a camera take for a preconfigured time period, removing the camera take from the registered camera pool.

10. The method according to claim 6 , wherein:
when a time period that the channel has no new frame selection is less than a preset threshold, the ranking score of the video frame is calculated according to number of inquiries of the channel and the similarity score between the video frame and the last selected frame of the corresponding camera take;
when the last selected frame of the corresponding camera take does not exist, the similarity score is zero; and
when a time period that the channel has no new frame selection is greater than a preset threshold, the ranking score of the video frame is a constant.

11. A video program recognition system, comprising:
a recorder configured to record video sequences of each of a plurality of channels, wherein the video sequences of each of the plurality of channels are formed by interleaved segments of camera takes filmed with a plurality of original camera setups, and a camera take is a series of frames filmed with one camera setup;
a server having one or more processors configured to:
obtain a set of features of an image captured by a first user device of a video frame from one of the plurality of channels and played on a second user device different from the first user device, wherein the first user device includes a communication module configured to transmit the set of features of the image captured by the first user device to the server;
perform a camera registration operation on the video sequences of each of the plurality of channels to select a plurality of video frames representing the recorded video sequences of each of the plurality of channels based on key video frames from a plurality of registered cameras associated with the recorded video sequences from the recorder; and
extract features of at least some of the selected plurality of video frames of each of the plurality of channels; and

a feature database configured to store the extracted features of each of the plurality of channels from the server,
wherein the server is further configured to:
compare the obtained set of features of the image with the stored features of each of the plurality of channels to determine whether there is a match between the obtained set of features of the image and features of a video frame from a specific channel; and
when there is a match between the obtained set of features of the image and features of a video frame from a specific channel, determine a channel number of the specific channel as a recognized channel number;

and wherein, to perform the camera registra

Assignees

• Tcl Res America Inc

Inventors

• Wang Haohong
• Ren Xiaobo
• Yang Yimin

Classifications

• G06V20/46

  Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames · CPC title

• G06V10/772

  Determining representative reference patterns, e.g. averaging or distorting patterns; Generating dictionaries · CPC title

• G06V10/462

  Salient features, e.g. scale invariant feature transforms [SIFT] · CPC title

• G06V10/95

  structured as a network, e.g. client-server architectures · CPC title

• G06V10/75

  Organisation of the matching processes, e.g. simultaneous or sequential comparisons of image or video features; Coarse-fine approaches, e.g. multi-scale approaches; using context analysis; Selection of dictionaries · CPC title