Method for processing video data, electronic device and computer storage

Patent metadata

Field	Value
Publication number	US-12530889-B2
Application number	US-202218084444-A
Country	US
Kind code	B2
Filing date	Dec 19, 2022
Priority date	Dec 23, 2021
Publication date	Jan 20, 2026
Grant date	Jan 20, 2026

Abstract

A method for processing video data includes: acquiring a target segment of video data to be extracted; acquiring theme information to be extracted; and determining an association degree between the target segment and the theme information based on segment information of the video data and a relationship between the target segment and the video data.

First claim

What is claimed is:

1 . A method for processing video data, comprising:
acquiring a target segment of video data to be extracted; acquiring theme information to be extracted; and determining an association degree between the target segment and the theme information based on segment information of the video data and a relationship between the target segment and the video data; wherein the association degree is determined by using a graph neural network.

2 . The method of claim 1 , wherein determining the association degree between the target segment and the theme information based on the segment information of the video data and the relationship between the target segment and the video data comprises:
determining a plurality of theme content segments based on the theme information; determining a relationship among the plurality of theme content segments; and determining the association degree between the target segment and the theme information based on the plurality of theme content segments, the relationship among the plurality of theme content segments, the segment information of the video data and the relationship between the target segment and the video data.

3 . The method of claim 2 , wherein determining the association degree between the target segment and the theme information based on the plurality of theme content segments, the relationship among the plurality of theme content segments, the segment information of the video data and the relationship between the target segment and the video data comprises:
obtaining segment features of the plurality of theme content segments, a segment feature of a video segment, a global feature of the video segment and a global feature of the plurality of theme content segments by encoding the plurality of theme content segments, the relationship among the plurality of theme content segments, the segment information of the video data and the relationship between the target segment and the video data; obtaining a code of the video segment by repeatedly performing encoding up to a set number of times based on the segment features of the plurality of theme content segments, the segment feature of the video segment, the global feature of the video segment, the global feature of the plurality of theme content segments and position information of the plurality of theme content segments; and determining an association degree between the video segment and the theme information based on the code of the video segment.

4 . The method of claim 3 , wherein determining the association degree between the video segment and the theme information based on the code of the video segment comprises:
acquiring a determination result by performing conditional random field (CRF) determination on the code of the video segment; and determining the association degree between the video segment and the theme information based on the determination result.

5 . The method of claim 1 , wherein acquiring the target segment of the video data to be extracted comprises:
obtaining a plurality of video segments of the video data to be extracted by dividing the video data based on a set interval; and taking at least one of the plurality of video segments of the video data as the target segment.

6 . The method of claim 1 , wherein the segment information comprises a feature of a video segment, and the method further comprises:
converting each video frame in the video segment to a vector of a set dimension, the vector of the set dimension comprising content information of a corresponding video frame; and determining the feature of the video segment based on the vector of the set dimension.

7 . The method of claim 6 , wherein determining the feature of the video segment based on the vector of the set dimension comprises:
obtaining a spatio-temporal information feature of a video frame by performing 3-dimensional convolutional neural network (C3D) determination on the vector of the set dimension; and determining the feature of the video segment based on the spatio-temporal information feature of the video frame.

8 . The method of claim 6 , wherein determining the feature of the video segment based on the vector of the set dimension comprises:
determining an optical flow (OF) feature of the video segment based on switching information between video frames of the video segment; and determining the feature of the video segment based on the vector of the set dimension and the OF feature.

9 . The method of claim 8 , wherein determining the feature of the video segment based on the vector of the set dimension and the OF feature comprises:
obtaining a spliced feature by splicing the vector of the set dimension, the spatio-temporal information feature of the video frame and the OF feature; and determining the feature of the video segment based on the spliced feature.

10 . The method of claim 9 , wherein determining the feature of the video segment based on the spliced feature comprises:
determining a feature of each video frame in the video segment based on a spliced feature of a previous video frame of the each video frame.

11 . The method of claim 1 , wherein the relationship between the target segment and the video data comprises a relationship between the target segment and a non-target segment in the video data, and time information of the target segment in the video data.

12 . The method of claim 1 , wherein the graph neural network comprises a theme information graph neural network and a video segment graph neural network; and determining the association degree comprises:
determining a global feature of a theme content segment comprised in the theme information based on the theme content segment using the theme information graph neural network; determining a global feature of a video segment comprised in the video data based on the segment information of the video data and the relationship between the target segment and the video data using the video segment graph neural network; and determining the association degree based on the global feature of the theme content segment and the global feature of the video segment comprised in the video data.

13 . The method of claim 1 , further comprising:
determining a video abstract of the video data based on the association degree.

14 . An electronic device, comprising:
at least one processor; and a memory communicatively connected to the at least one processor; wherein, the memory is stored with instructions executable by the at least one processor, and the instructions, when executed by the at least one processor, cause the at least one processor to perform a method for processing video data, comprising: acquiring a target segment of video data to be extracted; acquiring theme information to be extracted; and determining an association degree between the target segment and the theme information based on segment information of the video data and a relationship between the target segment and the video data; wherein the association degree is determined by using a graph neural network.

15 . The electronic device of claim 14 , wherein determining the association degree between the target segment and the theme information based on the segment information of the video data and the relationship between the target segment and the video data comprises:
determining a plurality of theme content segments based on the theme information; determining a relationship among the plurality of theme content segments; and determining the association degree between the target segment and the theme information based o

Assignees

• Beijing Baidu Netcom Sci & Tech Co Ltd

Inventors

• Bian Donghai
• Wu Yuwei
• Sheng Guangzhi
• Zheng Yehan
• Peng Weihua
• Xu Weijian

Classifications

• G06V10/82

  using neural networks · CPC title

• G06V10/7715

  Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods · CPC title

• G06V20/49

  Segmenting video sequences, i.e. computational techniques such as parsing or cutting the sequence, low-level clustering or determining units such as shots or scenes · CPC title

• G06T7/246

  using feature-based methods, e.g. the tracking of corners or segments · CPC title

• G06T7/215

  Motion-based segmentation · CPC title