Graph convolutional networks for video grounding

Patent metadata

Field	Value
Publication number	US-11442986-B2
Application number	US-202016792208-A
Country	US
Kind code	B2
Filing date	Feb 15, 2020
Priority date	Feb 15, 2020
Publication date	Sep 13, 2022
Grant date	Sep 13, 2022

Abstract

Method and apparatus that includes receiving a query describing an aspect in a video, the video including a plurality of frames, identifying multiple proposals that potentially correspond to the query where each of the proposals includes a subset of the plurality of frames, ranking the proposals using a graph convolution network that identifies relationships between the proposals, and selecting, based on the ranking, one of the proposals as a video segment that correlates to the query.

First claim

What is claimed is:

1. A method comprising:
receiving a query describing an aspect in a video, the video comprising a plurality of frames;
identifying multiple proposals that potentially correspond to the query, wherein each of the proposals comprises a subset of the plurality of frames;
generating a graph based on the query and the multiple proposals;
ranking the proposals using a graph convolution network (GCN) that identifies relationships between the proposals, wherein the graph is input into the graph convolution network; and
selecting, based on the ranking, one of the proposals as a video segment that correlates to the query.

2. The method of claim 1 , wherein generating the graph comprises:
identifying visual features in the proposals using a visual feature encoder; and
generating query features from the query using a recurrent neural network (RNN).

3. The method of claim 2 ,
wherein the graph comprises nodes and edges based on the visual features and the query features.

4. The method of claim 3 , wherein ranking the proposals comprises:
updating node features for the nodes in the graph; and
calculating edge weights for the edges in the graph.

5. The method of claim 3 , wherein ranking the proposals further comprises:
performing node aggregation; and
ranking the proposals based on the node aggregation and results from processing the graph using the GCN.

6. The method of claim 1 , wherein at least two proposals of the multiple proposals comprise overlapping frames of the plurality of frames in the video.

7. The method of claim 1 , wherein at least two proposals of the multiple proposals comprise subsets of the plurality of frames in the video that do not overlap.

8. A system, comprising:
a processor; and
memory comprising a program, which when executed by the processor performs an operation, the operation comprising:
receiving a query describing an aspect in a video, the video comprising a plurality of frames;
identifying multiple proposals that potentially correspond to the query, wherein each of the proposals comprises a subset of the plurality of frames;
generating a graph based on the query and the multiple proposals;
ranking the proposals using a GCN that identifies relationships between the proposals, wherein the graph is input into the graph convolution network; and
selecting, based on the ranking, one of the proposals as a video segment that correlates to the query.

9. The system of claim 8 , wherein generating the graph comprises:
identifying visual features in the proposals using a visual feature encoder; and
generating query features from the query using a recurrent neural network (RNN).

10. The system of claim 9 ,
wherein the graph comprises nodes and edges based on the visual features and the query features.

11. The system of claim 10 , wherein ranking the proposals comprises:
updating node features for the nodes in the graph; and
calculating edge weights for the edges in the graph.

12. The system of claim 10 , wherein ranking the proposals further comprises:
performing node aggregation; and
ranking the proposals based on the node aggregation and results from processing the graph using the GCN.

13. The system of claim 8 , wherein at least two proposals of the multiple proposals comprise overlapping frames of the plurality of frames in the video.

14. The system of claim 8 , wherein at least two proposals of the multiple proposals comprise subsets of the plurality of frames in the video that do not overlap.

15. A computer program product for identifying a video segment that correlates to a query, the computer program product comprising:
a computer readable storage medium having computer readable program code embodied therewith, the computer-readable program code executable by one or more computer processors to perform an operation, the operation comprising:
receiving the query, the query describing an aspect in a video comprising a plurality of frames;
identifying multiple proposals that potentially correspond to the query, wherein each of the proposals comprises a subset of the plurality of frames;
generating a graph based on the query and the multiple proposals;
ranking the proposals using a GCN that identifies relationships between the proposals, wherein the graph is input into the graph convolution network; and
selecting, based on the ranking, one of the proposals as the video segment that correlates to the query.

16. The computer program product of claim 15 , wherein generating the graph comprises:
identifying visual features in the proposals using a visual feature encoder; and
generating query features from the query using a recurrent neural network (RNN).

17. The computer program product of claim 16 ,
wherein the graph comprises nodes and edges based on the visual features and the query features.

18. The computer program product of claim 17 , wherein ranking the proposals comprises:
updating node features for the nodes in the graph; and
calculating edge weights for the edges in the graph.

19. The computer program product of claim 17 , wherein ranking the proposals further comprises:
performing node aggregation; and
ranking the proposals based on the node aggregation and results from processing the graph using the GCN.

20. The computer program product of claim 15 , wherein at least two proposals of the multiple proposals comprise overlapping frames of the plurality of frames in the video.

Assignees

• IBM

Inventors

• Gan Chuang
• Liu Sijia
• Das Subhro
• Wang Dakuo
• Zhang Yang

Classifications

• G06F16/7837 Primary

  using objects detected or recognised in the video content · CPC title

• G06V20/41

  Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items (segmenting video sequences G06V20/49) · CPC title

• G06V10/7784

  based on feedback from supervisors · CPC title

• G06V10/82

  using neural networks · CPC title

• G06V10/764

  using classification, e.g. of video objects · CPC title