Keypoint based action localization

Patent metadata

Field	Value
Publication number	US-12198397-B2
Application number	US-202217586284-A
Country	US
Kind code	B2
Filing date	Jan 27, 2022
Priority date	Jan 28, 2021
Publication date	Jan 14, 2025
Grant date	Jan 14, 2025

Abstract

A computer-implemented method is provided for action localization. The method includes converting one or more video frames into person keypoints and object keypoints. The method further includes embedding position, timestamp, instance, and type information with the person keypoints and object keypoints to obtain keypoint embeddings. The method also includes predicting, by a hierarchical transformer encoder using the keypoint embeddings, human actions and bounding box information of when and where the human actions occur in the one or more video frames.

First claim

What is claimed is:

1. A computer-implemented method for action localization, comprising:
converting one or more video frames into person keypoints and object keypoints;
embedding position, timestamp, instance, and type information with the person keypoints and object keypoints to obtain keypoint embeddings; and
predicting, by a hierarchical transformer encoder using the keypoint embeddings, human actions and bounding box information of when and where the human actions occur in the one or more video frames, the embedding including converting the keypoints to tokens, and the predicting including projecting the tokens to embedding metrics and summing the embedding metrics to obtain an output keypoint embedding.

2. The computer-implemented method of claim 1 , wherein said converting converts the one or more video frames into person keypoints in a form of human joint names for each detected person.

3. The computer-implemented method of claim 2 , wherein said converting further comprises selecting a top N out of detected persons based on person detection confidence scores.

4. The computer-implemented method of claim 1 , wherein said converting comprises extracting the object keypoints by subsampling a contour of an object mask detected by a Mask R-CNN.

5. The computer-implemented method of claim 4 , wherein said converting further comprises selecting top N out of detected objects based on object detection confidence scores.

6. The computer-implemented method of claim 1 , further comprising learning atomic actions from the person keypoints and the object keypoints.

7. The computer-implemented method of claim 1 , wherein the position information comprises a down-sampled spatial location of each pixel coordinate.

8. The computer-implemented method of claim 1 , wherein the timestamp information comprises a difference between a keypoint timestamp and a beginning keyframe timestamp.

9. The computer-implemented method of claim 1 , wherein the instance information comprises a spatial correlation between the person keypoints and a person instance.

10. The computer-implemented method of claim 1 , wherein the type information comprises a human body part name.

11. The computer-implemented method of claim 1 , wherein the position, timestamp, instance, and type information comprise representative tokens that are linearly projected to a respective embedding metric and summed to obtain an output keypoint embedding through a transformer based Keypoint Embedding Network.

12. The computer-implemented method of claim 1 , further comprising controlling a vehicle system for accident avoidance responsive to the predicted human actions and the bounding box information.

13. The computer-implemented method of claim 1 , further comprising controlling a robotic system for collision avoidance responsive to the predicted human actions and the bounding box information.

14. A computer program product for action localization, the computer program product comprising a non-transitory computer readable storage medium having program instructions embodied therewith, the program instructions executable by a computer to cause the computer to perform a method comprising:
converting, by a processor device of the computer, one or more video frames into person keypoints and object keypoints;
embedding, by the processor device, position, timestamp, instance, and type information with the person keypoints and object keypoints to obtain keypoint embeddings; and
predicting, by a hierarchical transformer encoder of the computer using the embedded keypoints, human actions and bounding box information of when and where the human actions occur in the one or more video frames, the embedding including converting the keypoints to tokens, and the predicting including projecting the tokens to embedding metrics and summing the embedding metrics to obtain an output keypoint embedding.

15. The computer program product of claim 14 , wherein said converting converts the one or more video frames into person keypoints in a form of human joint names for each detected persons.

16. The computer program product of claim 15 , wherein said converting further comprises selecting a top N out of the detected persons based on person detection confidence scores.

17. The computer program product of claim 14 , wherein said converting comprises extracting the object keypoints by subsampling a contour of a mask detected by a Mask R-CNN.

18. The computer program product of claim 17 , wherein said converting further comprises selecting top N out of detected objects based on object detection confidence scores.

19. The computer program product of claim 14 , further comprising learning atomic actions in the person keypoints and the object keypoints.

20. A computer processing system for action localization, comprising:
a memory device for storing program code;
a processor device operatively coupled to the memory device for running the program code for:
converting one or more video frames into person keypoints and object keypoints;
embedding position, timestamp, instance, and type information with the person keypoints and object keypoints to obtain keypoint embeddings; and
predicting, using a hierarchical transformer encoder that inputs the keypoint embeddings, human actions and bounding box information of when and where the human actions occur in the one or more video frames, the embedding including converting the keypoints to tokens, and the predicting including projecting the tokens to embedding metrics and summing the embedding metrics to obtain an output keypoint embedding.

Assignees

• Nec Lab America Inc
• Nec Corp

Inventors

• Kadav Asim
• Lai Farley
• Graf Hans Peter
• Huang Yi

Classifications

• G06V10/44

  Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components · CPC title

• G08G1/166

  for active traffic, e.g. moving vehicles, pedestrians, bikes · CPC title

• G06V20/58

  Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads · CPC title

• G06V10/82

  using neural networks · CPC title

• G06T2207/30261

  Obstacle · CPC title