Method and apparatus for matching of corresponding frames in multimedia streams

Patent metadata

Field	Value
Publication number	US-9584844-B2
Application number	US-201414547100-A
Country	US
Kind code	B2
Filing date	Nov 18, 2014
Priority date	Nov 21, 2013
Publication date	Feb 28, 2017
Grant date	Feb 28, 2017

Abstract

A method and an apparatus for matching corresponding frames in two multimedia streams are suggested. The first multimedia stream is an original multimedia stream for which meta-data are available. The second multimedia stream is broadcasted or multi-casted multimedia stream and has undergone processing steps such that the meta-data of the original video stream are no more available or no longer reliable. The matching of corresponding frames in the two multimedia streams is based on the comparison of audio fingerprints and the detection of shot boundaries. Matching corresponding frames with frame accuracy enables inserting targeted content like advertisements into a sequence of video frames especially in a broadcast or multicast environment where meta-data and time markers cannot be relied upon.

First claim

The invention claimed is:

1. Method for matching a corresponding video frame in a first and a second multimedia streams containing multimedia data, wherein the multimedia data include at least a video stream and an audio stream, wherein the first multimedia stream is provided with meta-data, and wherein the method comprises:
receiving the second multimedia stream;
receiving the meta-data and audio-fingerprints of the first multimedia stream;
detecting a video shot boundary in the second multimedia stream and a first video frame following the video shot boundary in the second multimedia stream;
determining the audio-fingerprint of a sequence of video frames including the video shot boundary and the first video frame after the video shot boundary has been detected in the second multimedia stream;
matching the determined audio-fingerprint with an audio-fingerprint of the first multimedia stream, wherein the audio-finger print matching is used to determine coarse synchronization of a sequence of corresponding video frames in the first and the second multimedia streams, the coarse synchronization does not satisfy frame accurate synchronization of the corresponding video frame;
determining a video sequence in the first multimedia stream corresponding to the matching audio-fingerprint to provide the coarse synchronization;
detecting a video shot boundary in the video sequence in the first multimedia stream; and
identifying a second video frame following the video shot boundary in the video sequence, wherein the second video frame matches with the first video frame to provide the frame accurate synchronization of the corresponding video frame.

2. Method according to claim 1 , wherein the method further comprises inserting content into the second multimedia stream.

3. Method according to claim 2 , wherein the method further comprises requesting content from a server before inserting the content into the second multimedia stream.

4. Method according to claim 3 , wherein the method further comprises
storing information about user behavior; and
inserting content which is aligned with the information about user behavior.

5. Method according to claim 2 , wherein inserting content comprises replacing at least a portion of the second multimedia stream by other content.

6. Method according to claim 2 , wherein inserting content comprises replacing a plurality of video frames as a whole by other video frames.

7. Method according to claim 2 , inserting the content is executed on a server and/or on a cloud computer.

8. Apparatus for matching a corresponding video frame in a first and a second multimedia streams comprising
a broadcast signal receiver configured to receive the first multimedia stream transmitted with meta-data and audio-fingerprints of the first multimedia stream, wherein the received transmitted first multimedia stream at the receiver being a second multimedia stream; and
a processor configured to detect a video shot boundary in the second multimedia stream and a first video frame following the video shot boundary in the second multimedia stream;
to determine the audio-fingerprint of a sequence of frames including the shot boundary and the first video frame after the shot boundary has been detected in the second multimedia stream;
to match the determined audio-fingerprint with an audio-fingerprint of the first multimedia stream, wherein the audio-finger print matching is used to determine coarse synchronization of a sequence of corresponding video frames in the first and the second multimedia streams, the coarse synchronization does not satisfy frame accurate synchronization of the corresponding video frame;
to determine a video sequence in the first multimedia stream corresponding to the matching audio-fingerprint to provide the coarse synchronization;
to detect a video shot boundary in the video sequence in the first multimedia stream; and
to identify a second video frame following the video shot boundary in the video sequence, wherein the second video frame matches with the first video frame to provide the frame accurate synchronization of the corresponding video frame.

9. Apparatus according to claim 8 , wherein the processor is adapted to accumulate information about viewer behavior.

10. Apparatus according to claim 8 , wherein the apparatus is equipped with communication interface adapted to request and to receive information about viewer behavior from an external source.

11. Apparatus according to claim 9 , wherein the apparatus is equipped with communication interface adapted to request and to receive information about viewer behavior from an external source.

12. Apparatus according to claim 9 , wherein the apparatus includes a memory for storing the accumulated and/or received information about viewer behavior.

13. Apparatus according to claim 10 , wherein the apparatus includes a memory for storing the accumulated and/or received information about viewer behavior.

14. Apparatus according to claim 9 , wherein the apparatus includes a memory for storing information about a plurality of viewers.

15. Apparatus according to claim 10 , wherein the apparatus includes a memory for storing information about a plurality of viewers.

16. A computer program product stored in non-transitory computer-readable storage media for matching a corresponding frame in a first and a second multimedia streams containing multimedia data, wherein the multimedia data include at least a video stream and an audio stream, wherein the first multimedia stream is provided with meta-data, comprising computer-executable instructions for a television receiver, wherein the instructions comprising:
receiving the second multimedia stream;
receiving the meta-data and audio-fingerprints of the first multimedia stream;
detecting a video shot boundary in the second multimedia stream and a first video frame following the video shot boundary in the second multimedia stream;
determining the audio-fingerprint of a sequence of video frames including the video shot boundary and the first video frame after the video shot boundary has been detected in the second multimedia stream;
matching the determined audio-fingerprint with an audio-fingerprint of the first multimedia stream, wherein the audio-finger print matching is used to determine coarse synchronization of a sequence of corresponding video frames in the first and the second multimedia streams, the coarse synchronization does not satisfy frame accurate synchronization of the corresponding video frame;
determining a video sequence in the first multimedia stream corresponding to the matching audio-fingerprint to provide the coarse synchronization;
detecting a video shot boundary in the video sequence in the first multimedia stream; and
identifying a second video frame following the video shot boundary in the video sequence, wherein the second video frame matches with the first video frame to provide the frame accurate synchronization of the corresponding video frame.

Assignees

• Thomson Licensing
• Thomson Licensing Sas

Inventors

• Onno Stephane
• Neumann Christoph
• Defrance Serge

Classifications

• H04N21/812

  involving advertisement data (advertising per se G06Q30/02) · CPC title

• H04N21/8358

  involving watermark {(protecting executable software by watermarking G06F21/16; image watermarking in general G06T1/0021; watermarks inserted in still images for transmission purposes H04N1/32144; inserting watermarks during video coding H04N19/467)} · CPC title

• H04N21/4394

  involving operations for analysing the audio stream, e.g. detecting features or characteristics in audio streams (arrangements characterised by components specially adapted for monitoring, identification or recognition of audio in broadcast systems H04H60/58) · CPC title

• H04N21/4307

  Synchronising the rendering of multiple content streams or additional data on devices, e.g. synchronisation of audio on a mobile phone with the video output on the TV screen · CPC title

• H04N21/44016

  involving splicing one content stream with another content stream, e.g. for substituting a video clip · CPC title