Video frame action detection using gated history

Patent metadata

Field	Value
Publication number	US-11895343-B2
Application number	US-202217852310-A
Country	US
Kind code	B2
Filing date	Jun 28, 2022
Priority date	Jun 3, 2022
Publication date	Feb 6, 2024
Grant date	Feb 6, 2024

Abstract

Example solutions for video frame action detection use a gated history and include: receiving a video stream comprising a plurality of video frames; grouping the plurality of video frames into a set of present video frames and a set of historical video frames, the set of present video frames comprising a current video frame; determining a set of attention weights for the set of historical video frames, the set of attention weights indicating how informative a video frame is for predicting action in the current video frame; weighting the set of historical video frames with the set of attention weights to produce a set of weighted historical video frames; and based on at least the set of weighted historical video frames and the set of present video frames, generating an action prediction for the current video frame.

First claim

What is claimed is:

1. A system comprising:
a processor; and
a computer-readable medium storing instructions that are operative upon execution by the processor to:
receive a video stream comprising a plurality of video frames;
group the plurality of video frames into a set of present video frames and a set of historical video frames, the set of present video frames comprising a current video frame;
determine a set of attention weights for the set of historical video frames, the set of attention weights indicating how informative a video frame is for predicting action in the current video frame;
weight the set of historical video frames with the set of attention weights to produce a set of weighted historical video frames; and
based on at least the set of weighted historical video frames and the set of present video frames, generate an action prediction for the current video frame.

2. The system of claim 1 , wherein the instructions are further operative to:
based on at least the action prediction for the current video frame, generate an annotation for the current video frame; and
display the current video frame subject to the annotation for the current video frame.

3. The system of claim 1 , wherein determining the set of attention weights comprises:
determining, for each video frame of the set of historical video frames, a position-guided gating score.

4. The system of claim 1 , wherein the plurality of video frames comprises a set of history frames and, for each individual history frame in the set of history frames, a set of subsequently-observed video frames, wherein the set of subsequently-observed video frames is more recent than the individual history frame, and wherein the instructions are further operative to:
based on at least the set of history frames and their sets of subsequently-observed video frames, extract features from the set of historical video frames; and
encode the extracted features.

5. The system of claim 4 , wherein extracting features does not use optical flow.

6. The system of claim 1 , wherein the instructions are further operative to:
perform background suppression, wherein the action prediction comprises a confidence and wherein performing the background suppression comprises:
modifying the confidence, including by weighting low confidence video frames more heavily, with separate emphasis on action and background classes, for a classifier that generates the action prediction.

7. The system of claim 1 , wherein the action prediction comprises a no action prediction or an action class prediction selected from a plurality of action classes.

8. A computerized method comprising:
receiving a video stream comprising a plurality of video frames;
grouping the plurality of video frames into a set of present video frames and a set of historical video frames, the set of present video frames comprising a current video frame;
determining a set of attention weights for the set of historical video frames, the set of attention weights indicating how informative a video frame is for predicting action in the current video frame;
weighting the set of historical video frames with the set of attention weights to produce a set of weighted historical video frames; and
based on at least the set of weighted historical video frames and the set of present video frames, generating an action prediction for the current video frame.

9. The method of claim 8 , further comprising:
based on at least the action prediction for the current video frame, generating an annotation for the current video frame; and
displaying the current video frame subject to the annotation for the current video frame.

10. The method of claim 8 , wherein determining the set of attention weights comprises:
determining, for each video frame of the set of historical video frames, a position-guided gating score.

11. The method of claim 8 , wherein the plurality of video frames comprises a set of history frames and, for each history frame in the set of history frames, a set of subsequently-observed video frames, wherein the set of subsequently-observed video frames is more recent than the history frame, and wherein the method further comprises:
based on at least the set of history frames and their sets of subsequently-observed video frames, extracting features from the set of historical video frames; and
encoding the extracted features.

12. The method of claim 11 , wherein extracting features does not use optical flow.

13. The method of claim 8 , further comprising:
performing background suppression, wherein the action prediction comprises a confidence and wherein performing the background suppression comprises:
weighting low confidence video frames more heavily, with separate emphasis on action and background classes, for a classifier that generates the action prediction.

14. The method of claim 8 , wherein the action prediction comprises a no action prediction or an action class prediction selected from a plurality of action classes.

15. One or more computer storage devices having computer-executable instructions stored thereon, which, on execution by a computer, cause the computer to perform operations comprising:
receiving a video stream comprising a plurality of video frames;
grouping the plurality of video frames into a set of present video frames and a set of historical video frames, the set of present video frames comprising a current video frame;
determining a set of attention weights for the set of historical video frames, the set of attention weights indicating how informative a video frame is for predicting action in the current video frame;
weighting the set of historical video frames with the set of attention weights to produce a set of weighted historical video frames; and
based on at least the set of weighted historical video frames and the set of present video frames, generating an action prediction for the current video frame.

16. The one or more computer storage devices of claim 15 , wherein the operations further comprise:
based on at least the action prediction for the current video frame, generating an annotation for the current video frame; and
displaying the current video frame subject to the annotation for the current video frame.

17. The one or more computer storage devices of claim 15 , wherein determining the set of attention weights comprises:
determining, for each video frame of the set of historical video frames, a position-guided gating score.

18. The one or more computer storage devices of claim 15 , wherein the plurality of video frames comprises a set of history frames and, for each history frame in the set of history frames, a set of subsequently-observed video frames, wherein the set of subsequently-observed video frames is more recent than the history frame, and wherein the operations further comprise:
based on at least the set of history frames and their sets of subsequently-observed video frames, extracting features from the set of historical video frames; and
encoding the extracted features.

19. The one or more computer storage devices of claim 15 , wherein the operations further comprise:
performing background suppression, wherein the action prediction comprises a confidence and wherein performing the background suppression comprises:
weighting low confidence video frames more heavily, with separate emphasis on action and background classes, for a classifier that generates the action prediction.

20. The one or more computer storage devices of cla

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Mittal Gaurav
• Yu Ye
• Chen Mei
• Chen Junwen

Classifications

• G06V20/52

  Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69) · CPC title

• G06V40/23

  Recognition of whole body movements, e.g. for sport training · CPC title

• G06V10/82

  using neural networks · CPC title

• G06V20/43

  of news video content · CPC title

• G06V20/46 Primary

  Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames · CPC title