Framework for few-shot temporal action localization

Patent metadata

Field	Value
Publication number	US-11727686-B2
Application number	US-202117481248-A
Country	US
Kind code	B2
Filing date	Sep 21, 2021
Priority date	Oct 23, 2019
Publication date	Aug 15, 2023
Grant date	Aug 15, 2023

Abstract

Systems and techniques that facilitate few-shot temporal action localization based on graph convolutional networks are provided. In one or more embodiments, a graph component can generate a graph that models a support set of temporal action classifications. Nodes of the graph can correspond to respective temporal action classifications in the support set. Edges of the graph can correspond to similarities between the respective temporal action classifications. In various embodiments, a convolution component can perform a convolution on the graph, such that the nodes of the graph output respective matching scores indicating levels of match between the respective temporal action classifications and an action to be classified. In various embodiments, an instantiation component can input into the nodes respective input vectors based on a proposed feature vector representing the action to be classified. In various cases, the respective temporal action classifications can correspond to respective example feature vectors, and the respective input vectors can be concatenations of the respective example feature vectors and the proposed feature vector.

First claim

What is claimed is:

1. A system, comprising:
a memory that stores computer-executable components; and
a processor, operably coupled to the memory, that executes the computer-executable components stored in the memory, wherein the computer-executable components comprise:
a graph component that generates a graph modeling a support set of temporal action classifications, such that nodes of the graph correspond to respective temporal action classifications in the support set; and
a convolution component that performs a convolution on the graph, such that the nodes output information indicating levels of match between the respective temporal action classifications and an action to be classified.

2. The system of claim 1 , further comprising:
an instantiation component that inputs into the nodes respective input vectors based on a proposed feature vector representing the action to be classified.

3. The system of claim 2 , wherein
the respective temporal action classifications correspond to respective example feature vectors; and
the respective input vectors are concatenations of the respective example feature vectors and the proposed feature vector representing the action to be classified.

4. The system of claim 3 , wherein:
the respective example feature vectors are generated via a first gated recurrent unit based on respective one-shot support videos that display exemplars of the respective temporal action classifications; and
the proposed feature vector is generated via a second gated recurrent unit based on an untrimmed input video that displays the action to be classified.

5. The system of claim 3 , wherein the similarities are calculated based on cosine similarities between respective example feature vectors of respective temporal action classifications.

6. The system of claim 1 , wherein the action to be classified is determined to belong to a temporal action classification having a highest matching score.

7. The system of claim 1 , wherein the graph is an undirected graph.

8. A computer-implemented method, comprising:
generating, by a device operatively coupled to a processor, a graph modeling a support set of temporal action classifications, such that nodes of the graph correspond to respective temporal action classifications in the support set; and
performing, by the device, a convolution on the graph, such that the nodes output information indicating levels of match between the respective temporal action classifications and an action to be classified.

9. The computer-implemented method of claim 8 , further comprising:
inputting, by the device, into the nodes respective input vectors based on a proposed feature vector representing the action to be classified.

10. The computer-implemented method of claim 9 , wherein
the respective temporal action classifications correspond to respective example feature vectors; and
the respective input vectors are concatenations of the respective example feature vectors and the proposed feature vector representing the action to be classified.

11. The computer-implemented method of claim 10 , further comprising:
generating, by the device, the respective example feature vectors via a first gated recurrent unit based on respective one-shot support videos that display exemplars of the respective temporal action classifications; and
generating, by the device, the proposed feature vector via a second gated recurrent unit based on an untrimmed input video that displays the action to be classified.

12. The computer-implemented method of claim 10 , wherein the similarities are calculated based on cosine similarities between respective example feature vectors of respective temporal action classifications.

13. The computer-implemented method of claim 8 , wherein the action to be classified is determined to belong to a temporal action classification having a highest matching score.

14. The computer-implemented method of claim 8 , wherein the graph is an undirected graph.

15. A computer program product for facilitating few-shot action localization, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a processing component to cause the processing component to:
generate a graph modeling a support set of temporal action classifications, such that first portions of the graph correspond to respective temporal action classifications in the support set, and such that second portions of the graph, distinct from the first portions, correspond to similarities between the respective temporal action classifications; and
perform a convolution on the graph, such that the nodes output information indicating levels of match between the respective temporal action classifications and an action to be classified.

16. The computer program product of claim 15 , wherein the program instructions are further executable to cause the processing component to:
input into the nodes respective input vectors based on a proposed feature vector representing the action to be classified.

17. The computer program product of claim 16 , wherein
the respective temporal action classifications correspond to respective example feature vectors; and
the respective input vectors are concatenations of the respective example feature vectors and the proposed feature vector representing the action to be classified.

18. The computer program product of claim 17 , wherein the program instructions are further executable to cause the processing component to:
generate the respective example feature vectors via a first gated recurrent unit based on respective one-shot support videos that display exemplars of the respective temporal action classifications; and
generate the proposed feature vector via a second gated recurrent unit based on an untrimmed input video that displays the action to be classified.

19. The computer program product of claim 17 , wherein the similarities are calculated based on cosine similarities between respective example feature vectors of respective temporal action classifications.

20. The computer program product of claim 15 , wherein the action to be classified is determined to belong to a temporal action classification having a highest matching score.

Assignees

• IBM

Inventors

• Gan Chuang
• Tan Ming
• Zhang Yang
• Wang Dakuo

Classifications

• G06N3/0442

  characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU] · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06V20/41 Primary

  Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items (segmenting video sequences G06V20/49) · CPC title

• G06F18/2323

  based on graph theory, e.g. minimum spanning trees [MST] or graph cuts · CPC title