Near-lossless video summarization

Patent metadata

Field	Value
Publication number	US-9628673-B2
Application number	US-76876910-A
Country	US
Kind code	B2
Filing date	Apr 28, 2010
Priority date	Apr 28, 2010
Publication date	Apr 18, 2017
Grant date	Apr 18, 2017

Abstract

Described is perceptually near-lossless video summarization for use in maintaining video summaries, which operates to substantially reconstruct an original video in a generally perceptually near-lossless manner. A video stream is summarized with little information loss by using a relatively very small piece of summary metadata. The summary metadata comprises an image set of synthesized mosaics and representative keyframes, audio data, and the metadata about video structure and motion. In one implementation, the metadata is computed and maintained (e.g., as a file) to summarize a relatively large video sequence, by segmenting a video shot into subshots, and selecting keyframes and mosaics based upon motion data corresponding to those subshots. The motion data is maintained as a semantic description associated with the image set. To reconstruct the video, the metadata is processed, including simulating motion using the image set and the semantic description, which recovers the audiovisual content without any significant information loss.

First claim

What is claimed is:

1. A method comprising:
extracting a plurality of subshots and a plurality of corresponding semantic descriptions from an original video;
selecting, for a subshot of the plurality of subshots, an image set based upon global motion data corresponding to the subshot, the global motion data identified using an extracted semantic description corresponding to the subshot;
selecting a keyframe within the image set based on the global motion data;
generating summarization metadata for the subshot, including by compressing the selected keyframe and maintaining the compressed keyframe, the extracted semantic description corresponding to the subshot associated with the image set, and data associated with the selected keyframe as the summarization metadata, such that the subshot is reconstructed by processing the summarization metadata.

2. The method of claim 1 wherein reconstructing the subshot further comprises simulating any motion based on the semantic description.

3. The method of claim 1 further comprising, outputting the subshot as part of a series of subshots to reconstruct the video.

4. The method of claim 1 wherein generating metadata for the subshot further comprises maintaining data associated with other frames in the image set with respect to the keyframe.

5. The method of claim 1 wherein generating metadata for the subshot further comprises maintaining an XML file having at least one reference to data in the image set.

6. The method of claim 1 wherein generating metadata for the subshot further comprises:
maintaining data representative of one or more of the following: a pan direction and magnitude, a tilt direction and magnitude, or a zoom direction and magnitude associated with the image set and the global motion data.

7. The method of claim 1 wherein generating metadata for the subshot further comprises:
obtaining an audio track from the original video;
compressing the obtained audio track; and
maintaining the compressed audio track in the summarization metadata.

8. The method of claim 1 wherein generating metadata for the subshot further comprises maintaining data representative of rotation direction and magnitude associated with the image set and the global motion data.

9. The method of claim 1 wherein generating metadata for the subshot further comprises maintaining data that indicates the subshot is static.

10. The method of claim 1 wherein generating metadata for the subshot further comprises maintaining object motion data associated with the image set.

11. The method of claim 10 wherein maintaining the object motion data further comprises determining object motion intensity, number of motion entities and object motion type.

12. The method of claim 10 wherein maintaining the object motion data further comprises determining whether the object background is static or dynamic.

13. A system comprising:
a subshot classification mechanism that extracts global motion data from frames of a video subshot to determine a motion type of that video subshot,
a subshot summarization mechanism that selects, based on the global motion data, an image set and at least one keyframe within the image set to be representative of the video subshot, compresses the at least one keyframe within the image set, and maintains the image set and data associated with the keyframe in association with a semantic description of the video subshot, including information corresponding to the global motion data, in summarization metadata representative of the video subshot; and
a video reconstruction mechanism that processes the summarization metadata to reconstruct the video subshot.

14. The system of claim 13 wherein the video reconstruction mechanism further processes the semantic description to reconstruct simulating any global motion of the video subshot and any object motion within the image set.

15. The system of claim 14 further comprising,
a composition mechanism that combines the reconstructed video subshot with at least one other reconstructed video subshot to reconstruct a video shot.

16. The system of claim 13 wherein the semantic description is maintained in a file with references to files having data corresponding to the image set, or wherein the semantic description is maintained in a file with data of the image set.

17. The system of claim 13 wherein the semantic description of the global motion data comprises data representative of one or more of the following: a pan direction and magnitude of the video subshot, a rotation direction and magnitude of the video subshot, a zoom direction and magnitude of the video subshot, or a tilt direction and magnitude of the video subshot.

18. The system of claim 13 wherein the semantic description comprises object motion data.

19. One or more computer storage devices having computer-executable instructions stored thereon for reconstructing a video, which upon execution by one or more processors, causes the one or more processors to perform operations comprising:
extracting a plurality of subshots and a plurality of corresponding semantic descriptions from an original video;
selecting, for respective subshots of the plurality of subshots, image sets based upon motion data corresponding to the respective subshots, the motion data identified using an extracted semantic description corresponding to a respective subshot;
selecting keyframes within the image sets based on the motion data;
generating summarization metadata for the respective subshots, including by compressing the selected keyframes within the images sets, the summarization metadata including the extracted semantic description corresponding to the respective subshot and data associated with the compressed keyframes, such that processing the generated summarization metadata using the motion data of the respective subshots to simulate at least one of any global motion or object motion reconstructs the respective subshots perceptually near-losslessly.

20. The one or more computer storage devices of claim 19 wherein the semantic descriptions comprise data representative of one or more of the following: a pan direction and magnitude of the respective subshots, a rotation direction and magnitude of the respective subshots, a zoom direction and magnitude of the respective subshots, a tilt direction and magnitude of the respective subshots.

Assignees

• Mei Tao
• Hua Xian-Sheng
• Li Shipeng
• Tang Lin-Xie
• Microsoft Technology Licensing Llc

Inventors

• Mei Tao
• Hua Xian-Sheng
• Li Shipeng
• Tang Lin-Xie

Classifications

• G06K9/00751

  Physics · mapped topic

• H04N19/25

  with scene description coding, e.g. binary format for scenes [BIFS] compression · CPC title

• H04N5/144 Primary

  Movement detection (for video coding H04N19/503; analysis of motion in general G06T7/20) · CPC title

• H04N21/8455

  involving pointers to the content, e.g. pointers to the I-frames of the video stream · CPC title

• H04N21/8453

  by locking or enabling a set of features, e.g. optional functionalities in an executable program · CPC title