Temporal contrastive learning for semi-supervised video action recognition

Patent metadata

Field	Value
Publication number	US-12067082-B2
Application number	US-202117515380-A
Country	US
Kind code	B2
Filing date	Oct 29, 2021
Priority date	Oct 29, 2021
Publication date	Aug 20, 2024
Grant date	Aug 20, 2024

Abstract

A base pathway of a computerized two-pathway video action recognition model is trained using a plurality of labeled video samples. The base pathway is trained using a plurality of unlabeled video samples at a first framerate. An auxiliary pathway of the computerized two-pathway video action recognition model is trained using a plurality of the unlabeled video samples at a second framerate, the second framerate being slower than the first framerate, wherein the training of the base pathway and the training of the auxiliary pathway result in a trained computerized two-pathway video action recognition model. A candidate video is categorized using the trained computerized two-pathway video action recognition model and the categorized candidate video is stored in a computer-accessible video database system for information retrieval.

First claim

What is claimed is:

1. A method comprising:
training a base pathway of a computerized two-pathway video action recognition model using a plurality of labeled video samples;
training the base pathway of the computerized two-pathway video action recognition model using a plurality of unlabeled video samples at a first framerate;
training an auxiliary pathway of the computerized two-pathway video action recognition model using a plurality of the unlabeled video samples at a second framerate, the second framerate being slower than the first framerate;
wherein said training of said base pathway using said plurality of labeled video samples, said training of said base pathway using said plurality of unlabeled video samples at said first framerate, and said training of said auxiliary pathway using said plurality of unlabeled video samples at said second framerate, result in a trained computerized two-pathway video action recognition model;
categorizing a candidate video using the trained computerized two-pathway video action recognition model; and
storing the categorized candidate video in a computer-accessible video database system for information retrieval.

2. The method of claim 1 , wherein the two-pathway video action recognition model comprises a temporal contrastive model.

3. The method of claim 1 , wherein a contrastive objective is based on a maximization of a similarity between encoded representations of a same video of the unlabeled video samples at different framerates and a minimization of a similarity between encoded representations of different videos of the unlabeled video samples at different speeds.

4. The method of claim 3 , wherein the similarity between the different videos of the unlabeled video samples is minimized by minimizing a modified Normalized Temperature-scaled Cross Entropy Loss (NT-Xent) contrastive loss between the different videos.

5. The method of claim 3 , further comprising forming groups of the unlabeled video samples having a same pseudo-label in a minibatch, and representing each group with an average representation of the unlabeled video samples within each group, wherein the contrastive objective is based on a group-contrastive loss between the groups of the unlabeled video samples that couples discriminative motion representation with pace-invariance.

6. The method of claim 1 , wherein the base pathway and the auxiliary pathway share a same set of weights.

7. The method of claim 6 , wherein the average representation is based on:

R
p
l

=

∑

i
=
1

B

𝕝

{

y
^

p
i

=
l

}

⁢

g
⁡
(

U
p
i

)

T

where ∥ is an indicator function that evaluates to 1 for videos with a pseudo-label equal to l∈Y in each pathway p∈{f, s}, g(U p i ) is a representation of a corresponding video, B is a count of videos in the minibatch, ŷ p i denotes pseudo-labels of the video U i , and T is a number of the videos with the pseudo-label equal to l∈Y in the minibatch.

8. The method of claim 1 , wherein the training operations are performed on a neural network backbone involving at least one of two-dimensional (2D) and three-dimensional (3D) convolution operations.

9. The method of claim 1 , further comprising minimizing a standard supervised cross-entropy loss ( sup ) on the labeled video samples, the standard supervised cross-entropy loss ( sup ) being given by:

ℒ
sup

=

-

∑

c
=
1

C

(

y
i

)

c

⁢

log
⁡
(

g
⁡
(

V
i

)

)

c

where g(V i ) is a representation of a corresponding video V i and C is a count of different activities.

10. The method of claim 1 , wherein the two-pathway video action recognition model is trained to match a representation g(U f i ) of a faster framerate version of a video (U i ) with a representation g(U s i ) of a comparatively slower framerate version of the video (U i ).

11. The method of claim 1 , wherein training the two-pathway video action recognition model is carried out using a loss function given by:
= sup γ* ic +β* gc

where sup is a standard supervised cross-entropy loss, ic is an instance-contrastive loss, gc is a group-contrastive loss, and γ and β are weights of the instance-contrastive and group-contrastive losses, respectively.

12. The method of claim 11 , wherein the instance-contrastive loss ♯ ic is:

ℒ
ic

(

U
f
i

,

U
s
i

)

=

Assignees

• IBM
• Indian Institute Of Tech Kharagpur
• Indian Institute Of Tech

Inventors

• Panda Rameswar
• Feris Rogerio Schmidt
• Das Abir

Classifications

• G06F18/24

  Classification techniques · CPC title

• G06F18/22

  Matching criteria, e.g. proximity measures · CPC title

• G06V20/41

  Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items (segmenting video sequences G06V20/49) · CPC title

• G06N3/08

  Learning methods · CPC title

• G06N3/04

  Architecture, e.g. interconnection topology · CPC title