Summarizing videos via side information

Patent metadata

Field	Value
Publication number	US-11538248-B2
Application number	US-202017081239-A
Country	US
Kind code	B2
Filing date	Oct 27, 2020
Priority date	Oct 27, 2020
Publication date	Dec 27, 2022
Grant date	Dec 27, 2022

Abstract

Machine learning-based techniques for summarizing collections of data such as image and video data leveraging side information obtained from related (e.g., video) data are provided. In one aspect, a method for video summarization includes: obtaining related videos having content related to a target video; and creating a summary of the target video using information provided by the target video and side information provided by the related videos to select portions of the target video to include in the summary. The side information can include video data, still image data, text, comments, natural language descriptions, and combinations thereof.

First claim

What is claimed is:

1. A method for video summarization, comprising:
obtaining related videos which, while being different videos from a target video, have content related to the target video; and
creating a summary of the target video using information provided by the target video and side information provided by the related videos to select which portions of the target video to use in creating the summary of the target video, wherein only the portions of the target video are included in the summary of the target video, and wherein the information provided by the target video and the side information provided by the related videos are used to create the summary of the target video by performing a consensus estimation on scored segments of the target video and the related videos such that, based on the consensus estimation, irrelevant segments of the target video are excluded from the summary of the target video.

2. The method of claim 1 , further comprising:
searching for the related videos online.

3. The method of claim 1 , wherein the side information comprises video data, still image data, text, comments, natural language descriptions, and combinations thereof.

4. The method of claim 1 , further comprising:
segmenting the target video and the related videos into segments;
representing the segments by feature vectors, wherein the feature vectors comprise three-dimensional convolutional neural network features extracted by performing temporal average pooling within each of the segments using a fixed number of input frames;
scoring the segments using the feature vectors to provide the scored segments of the target video and the related videos;
performing the consensus estimation of the scored segments of the target video and the related videos to obtain a unified set of the segments; and
generating the summary of the target video of a certain length from the unified set of the segments.

5. The method of claim 4 , wherein the target video and the related videos are segmented into multiple non-uniform segments.

6. The method of claim 5 , wherein the segmenting of the target video and the related videos into the segments comprises:
dividing the target video and each of the related videos into the multiple non-uniform segments by measuring an amount of change between two consecutive frames, and identifying a frame having a portion of total change that is greater than a predetermined threshold as a segment boundary.

7. The method of claim 4 , wherein v is the target video and {tilde over (v)} is a set of the related videos, wherein Y is a feature matrix of the target video v and {tilde over (Y)} is a feature matrix of the set of related videos, wherein n represents a total number of segments in the target video v and ñ represents a total number of segments in the set of the related videos {tilde over (v)}, and wherein the scoring of the segments using the feature vectors comprises:
obtaining an importance score of each of the segments in the target video v by solving,

min

C
,

C
~

⁢



Y
-
YC



F
2

+



Y
~

-

Y
⁢

C
~



F
2

+

α
⁡

(


C


1
,
2

+



C
~



1
,
2

)

over the feature matrix Y and the feature matrix {tilde over (Y)}, wherein C∈R n×n and {tilde over (C)}∈R n×ñ are score matrices and ∥C∥ 1,2 =Σ i=2 n ∥C i ∥ 2 , ∥C i ∥ 2 is an I 2 norm of an i-th row of C and indicates the importance score of an i-th video segment, and wherein α is a regularization parameter.

8. The method of claim 7 , wherein the performing of the consensus estimation comprises:
combining the score matrices C∈R n×n and {tilde over (C)}∈R n×ñ through a unified objective function as:

min

Y
,

Y
~

⁢



Y
-
YC



F
2

+



Y
~

-

Y
⁢

C
~



F
2

+

α
⁡

(


C


1
,
2

+



C
~



1
,

Assignees

• IBM

Inventors

• Panda Rameswar
• Gan Chuang
• Chen Pin-Yu
• Wu Bo

Classifications

• G06V20/47 Primary

  Detecting features for summarising video content · CPC title

• G06N20/00

  Machine learning · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G06V20/49

  Segmenting video sequences, i.e. computational techniques such as parsing or cutting the sequence, low-level clustering or determining units such as shots or scenes · CPC title

• G06V10/82

  using neural networks · CPC title