Systems and methods for multi-object tracking

Patent metadata

Field	Value
Publication number	US-12488477-B2
Application number	US-202318202632-A
Country	US
Kind code	B2
Filing date	May 26, 2023
Priority date	Jul 8, 2022
Publication date	Dec 2, 2025
Grant date	Dec 2, 2025

Abstract

A method for multiple object tracking includes receiving, with a computing device, a point cloud dataset, detecting one or more objects in the point cloud dataset, each of the detected one or more objects defined by points of the point cloud dataset and a bounding box, querying one or more historical tracklets for historical tracklet states corresponding to each of the one or more detected objects, implementing a 4D encoding backbone comprising two branches: a first branch configured to compute per-point features for each of the one or more objects and the corresponding historical tracklet states, and a second branch configured to obtain 4D point features, concatenating the per-point features and the 4D point features, and predicting, with a decoder receiving the concatenated per-point features, current tracklet states for each of the one or more objects.

First claim

What is claimed is:

1 . A method for multiple object tracking comprising:
receiving, with a computing device, a point cloud dataset; detecting one or more objects in the point cloud dataset, each of the detected one or more objects defined by points of the point cloud dataset and a bounding box; querying one or more historical tracklets for historical tracklet states corresponding to each of the one or more detected objects; implementing a 4D encoding backbone comprising two branches:
a first branch configured to compute per-point features for each of the one or more objects and the corresponding historical tracklet states, and
a second branch configured to obtain 4D point features;
concatenating the per-point features and the 4D point features; and predicting, with a decoder receiving the concatenated per-point features, current tracklet states for each of the one or more objects.

2 . The method of claim 1 , wherein the first branch comprises a PointNet architecture.

3 . The method of claim 1 , wherein the second branch comprises farthest point sampling and a local radii PointNet encoder to obtain the 4D point features PointNet architecture.

4 . The method of claim 1 , wherein the second branch comprises a set abstraction layer to subsample a set of anchor features in each frame of the point cloud dataset and a plurality of self-attention layers to generate updated anchor features.

5 . The method of claim 1 , wherein the decoder implemented to predict the current tracklet states comprises a plurality of separate decoding heads.

6 . The method of claim 5 , wherein at least one of the plurality of separate decoding heads comprises at least one of:
a first decoding head of the decoder configured to perform a per-point object-foreground-background segmentation, a second decoding head configured to regress per-frame object centers, a third decoding head configured to regresses per-frame object yaws, or a fourth decoding head configured to predict a single object size and a confidence score.

7 . The method of claim 1 , further comprising generating point cloud data of an environment with a LIDAR module communicatively coupled to the computing device, wherein the generated point cloud data forms the point cloud dataset received by the computing device.

8 . A system for multiple object tracking comprising:
a computing device, the computing device comprising a processor and a memory storing instructions that, when executed by the processor, cause the computing device to: receive a point cloud dataset; detect one or more objects in the point cloud dataset, each of the detected one or more objects defined by points of the point cloud dataset and a bounding box; query one or more historical tracklets for historical tracklet states corresponding to each of the one or more detected objects; implement a 4D encoding backbone comprising two branches:
a first branch configured to compute per-point features for each of the one or more objects and the corresponding historical tracklet states, and
a second branch configured to obtain 4D point features;
concatenate the per-point features and the 4D point features; and predict, with a decoder receiving the concatenated per-point features, current tracklet states for each of the one or more objects.

9 . The system of claim 8 , wherein the first branch comprises a PointNet architecture.

10 . The system of claim 8 , wherein the second branch comprises farthest point sampling and a local radii PointNet encoder to obtain the 4D point features PointNet architecture.

11 . The system of claim 8 , wherein the second branch comprises a set abstraction layer to subsample a set of anchor features in each frame of the point cloud dataset and a plurality of self-attention layers to generate updated anchor features.

12 . The system of claim 8 , wherein the decoder implemented to predict the current tracklet states comprises a plurality of separate decoding heads.

13 . The system of claim 12 , wherein at least one of the plurality of separate decoding heads comprises at least one of:
a first decoding head of the decoder configured to perform a per-point object-foreground-background segmentation, a second decoding head configured to regress per-frame object centers, a third decoding head configured to regresses per-frame object yaws, or a fourth decoding head configured to predict a single object size and a confidence score.

14 . The system of claim 8 , further comprising a LIDAR module, the LIDAR module communicatively coupled to the computing device and configured to generate point cloud data of an environment forming the point cloud dataset received by the computing device.

15 . A computing program product for multiple object tracking, the computing program product comprising machine-readable instructions stored on a non-transitory computer readable memory, which when executed by a computing device, causes the computing device to carry out steps comprising:
receiving, with the computing device, a point cloud dataset; detecting one or more objects in the point cloud dataset, each of the detected one or more objects defined by points of the point cloud dataset and a bounding box; querying one or more historical tracklets for historical tracklet states corresponding to each of the one or more detected objects; implementing a 4D encoding backbone comprising two branches:
a first branch configured to compute per-point features for each of the one or more objects and the corresponding historical tracklet states, and
a second branch configured to obtain 4D point features;
concatenating the per-point features and the 4D point features; and predicting, with a decoder receiving the concatenated per-point features, current tracklet states for each of the one or more objects.

16 . The computer program product of claim 15 , wherein the first branch comprises a PointNet architecture.

17 . The computer program product of claim 15 , wherein the second branch comprises farthest point sampling and a local radii PointNet encoder to obtain the 4D point features PointNet architecture.

18 . The computer program product of claim 15 , wherein the second branch comprises a set abstraction layer to subsample a set of anchor features in each frame of the point cloud dataset and a plurality of self-attention layers to generate updated anchor features.

19 . The computer program product of claim 15 , wherein the decoder implemented to predict the current tracklet states comprises a plurality of separate decoding heads.

20 . The computer program product of claim 19 , wherein at least one of the plurality of separate decoding heads comprises at least one of:
a first decoding head of the decoder configured to perform a per-point object-foreground-background segmentation, a second decoding head configured to regress per-frame object centers, a third decoding head configured to regresses per-frame object yaws, or a fourth decoding head configured to predict a single object size and a confidence score.

Assignees

• Toyota Res Inst Inc
• Toyota Motor Co Ltd
• Univ Leland Stanford Junior

Inventors

• Stearns Colton
• Li Jie
• Ambrus Rares A
• Guizilini Vitor Campagnolo
• Zakharov Sergey
• Gaidon Adrien D
• Rempe Davis
• Birdal Tolga
• Guibas Leonidas J

Classifications

• G06T2207/10028

  Range image; Depth image; 3D point clouds · CPC title

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title

• G06T2207/20081

  Training; Learning · CPC title

• G06T7/248 Primary

  involving reference images or patches · CPC title

• G06T7/246 Primary

  using feature-based methods, e.g. the tracking of corners or segments · CPC title