System and method for tracking detected objects

Patent metadata

Field	Value
Publication number	US-11775615-B2
Application number	US-202117242498-A
Country	US
Kind code	B2
Filing date	Apr 28, 2021
Priority date	Dec 4, 2020
Publication date	Oct 3, 2023
Grant date	Oct 3, 2023

Abstract

Systems and methods for tracking objects are disclosed herein. In one embodiment, a system having a processor merges features of detected objects extracted from a point cloud and a corresponding image to generate fused features for the detected objects, generates a learned distance metric for the detected objects using the fused features, determines matched detected objects and unmatched detected objects, applies prior tracking identifiers of the detected objects at the prior time to the matched detected objects, determines a confidence score for the fused features of the unmatched detected objects, and applies new tracking identifiers to the unmatched detected objects based on the confidence score.

First claim

What is claimed is:

1. A system for tracking detected objects comprising:
a processor;
a memory in communication with the processor, the memory having a feature fusion module, a distance combination module, a matching module, and a track initialization module;
the feature fusion module includes instructions that, when executed by the processor, cause the processor to merge features of detected objects extracted from a point cloud and a corresponding image to generate fused features for the detected objects;
the distance combination module includes instructions that, when executed by the processor, cause the processor to generate a learned distance metric for the detected objects using the fused features at a current time, fused features at a prior time, and Mahalanobis distances indicating distances between positions, orientations, and scales of detected objects at the current time and predicted positions, orientations, and scales of the detected objects determined at the prior time;
the matching module includes instructions that, when executed by the processor, cause the processor to determine matched detected objects and unmatched detected objects, wherein the matched detected objects are detected objects at the current time that have been matched with detected objects at the prior time based on the learned distance metric;
the matching module includes instructions that, when executed by the processor, cause the processor to apply prior tracking identifiers of the detected objects at the prior time to the matched detected objects; and
the track initialization module includes instructions that, when executed by the processor, cause the processor to determine a confidence score for the fused features of the unmatched detected objects and apply new tracking identifiers to the unmatched detected objects based on the confidence score.

2. The system of claim 1 , wherein the learned distance metric represents a similarity between the detected objects at the current time and the detected objects at the prior time.

3. The system of claim 1 , wherein the memory further includes a Kalman filter module having instructions that, when executed by the processor, cause the processor to:
output bounding boxes for the matched detected objects, the bounding boxes for the matched detected objects include positions, orientations, scales, and prior tracking identifiers; and
output bounding boxes for the unmatched detected objects that have been assigned new tracking identifiers, the bounding boxes for the unmatched detected objects including positions, orientations, scales, and new tracking identifiers.

4. The system of claim 3 , wherein the Kalman filter module further includes instructions that, when executed by the processor, cause the processor to determine predicted positions, orientations, scales of the detected objects based on positions, orientations, scales, angular velocities, and linear velocities of the detected objects.

5. The system of claim 1 , wherein the corresponding image includes bounding boxes for the detected objects projected from three-dimensional bounding boxes generated from the point cloud.

6. The system of claim 1 , wherein at least one of the feature fusion module, the distance combination module, and the track initialization module include one or more neural networks.

7. The system of claim 1 , wherein the fused features for the detected objects includes color information of the detected objects extracted from the corresponding image.

8. A method for tracking detected objects comprising the steps of:
merging features of detected objects extracted from a point cloud and a corresponding image to generate fused features for the detected objects;
generating a learned distance metric for the detected objects using the fused features at a current time, fused features at a prior time, and Mahalanobis distances indicating distances between positions, orientations, and scales of detected objects at the current time and predicted positions, orientations, and scales of the detected objects determined at the prior time;
determining matched detected objects and unmatched detected objects, wherein the matched detected objects are detected objects at the current time that have been matched with detected objects at the prior time based on the learned distance metric; and
applying prior tracking identifiers of the detected objects at the prior time to the matched detected objects.

9. The method of claim 8 , further comprising the step of outputting bounding boxes for the matched detected objects, the bounding boxes for the matched detected objects include positions, orientations, scales, and prior tracking identifiers.

10. The method of claim 8 , further comprising the step of determining predicted positions, orientations, scales of the detected objects based on positions, orientations, scales, angular velocities, and linear velocities of the detected objects.

11. The method of claim 8 , further comprising the steps of:
determining a confidence score for the fused features of the unmatched detected objects; and
applying new tracking identifiers to the unmatched detected objects based on the confidence score.

12. The method of claim 11 , further comprising the step of outputting bounding boxes for the unmatched detected objects that have been assigned new tracking identifiers, the bounding boxes for the unmatched detected objects including positions, orientations, scales, and new tracking identifiers.

13. The method of claim 8 , wherein the learned distance metric represents a similarity between the detected objects at the current time and the detected objects at the prior time.

14. The method of claim 8 , wherein the corresponding image includes bounding boxes for the detected objects projected from three-dimensional bounding boxes generated from the point cloud.

15. The method of claim 8 , wherein the fused features for the detected objects includes color information of the detected objects extracted from the corresponding image.

16. A non-transitory computer-readable medium having instructions that, when executed by a processor, cause the processor to:
merge features of detected objects extracted from a point cloud and a corresponding image to generate fused features for the detected objects;
generate a learned distance metric for the detected objects using the fused features at a current time, fused features at a prior time, and Mahalanobis distances indicating distances between positions, orientations, and scales of detected objects at the current time and predicted positions, orientations, and scales of the detected objects determined at the prior time;
determine matched detected objects and unmatched detected objects, wherein the matched detected objects are detected objects at the current time that have been matched with detected objects at the prior time based on the learned distance metric; and
apply prior tracking identifiers of the detected objects at the prior time to the matched detected objects.

17. The non-transitory computer-readable medium of claim 16 , further having instructions that, when executed by the processor, cause the processor to output bounding boxes for the matched detected objects, the bounding boxes for the matched detected objects include positions, orientations, scales, and prior tracking identifiers.

18. The non-transitory computer-readable medium of claim 16 , further having instructions that, when executed by the processor, cause the processor to:
determine a confidence score for the fused features of the unmatche

Assignees

• Toyota Res Inst Inc
• Univ Leland Stanford Junior

Inventors

• Chiu Hsu-Kuang
• Li Jie
• Ambrus Rares A
• Bohg Christin Jeannette

Classifications

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06F18/22 Primary

  Matching criteria, e.g. proximity measures · CPC title

• G06F18/213

  Feature extraction, e.g. by transforming the feature space; Summarisation; Mappings, e.g. subspace methods · CPC title

• G06F18/25

  Fusion techniques · CPC title