Methods and systems for object tracking using bounding boxes

Patent metadata

Field	Value
Publication number	US-10430641-B2
Application number	US-201715453444-A
Country	US
Kind code	B2
Filing date	Mar 8, 2017
Priority date	Mar 8, 2017
Publication date	Oct 1, 2019
Grant date	Oct 1, 2019

Abstract

Methods and systems are provided for tracking an object. The system includes a data receiving module configured to receive three dimensional imaging data and two dimensional imaging data. A three dimensional object identification module is configured to demarcate a three dimensional object and to output three-dimensional object data. An adjustment module is configured to adjust the three dimensional object data based on the two dimensional imaging data and to output adjusted three dimensional object data. A tracking module is configured to track at least one object using the adjusted three dimensional object data.

First claim

What is claimed is:

1. An object tracking system, comprising:
a data receiving module configured to, by a processor, receive three dimensional imaging data and two dimensional imaging data;
a three dimensional object identification module configured to, by a processor, identify a first bounding box that demarcates at least one three dimensional object within the three dimensional imaging data and to output three-dimensional object data;
a two dimensional object identification module configured to, by a processor, identify a second bounding box that demarcates at least one two dimensional object within the two dimensional imaging data and to output two-dimensional object data;
an adjustment module configured to, by a processor, perform bounding box regression based on the two dimensional object data to adjust a size and a dimension of the first bounding box and the second bounding box and to output adjusted three dimensional object data;
a tracking module configured to, by a processor, track at least one object using the adjusted three dimensional object data.

2. The object tracking system of claim 1 , wherein the adjustment module is configured to project the at least one three dimensional object into two dimensional imaging space of the two dimensional imaging data and to-perform the bounding box regression on the projected three dimensional imaging.

3. The object tracking system of claim 2 , wherein the adjustment module refines an aspect of the two dimensional object data based on the two dimensional imaging data and the bounding box regression.

4. The object tracking system of claim 3 , wherein the aspect represents number, dimensions and/or position of the at least one object in the two dimensional imaging data.

5. The object tracking system of claim 2 , wherein the adjustment module is configured to register the two dimensional imaging data with the three dimensional imaging data, and perform the projection on the registered two and three dimensional imaging data.

6. The object tracking system of claim 2 , wherein the adjustment module comprises a two dimensions to three dimensions transformation sub-module configured to transform the adjusted two dimensional object data to the adjusted three dimensional object data for use by the tracking module.

7. The object tracking system of claim 1 , wherein the adjustment module is configured to run a neural network to adjust the three dimensional object data based on the two dimensional imaging data.

8. The object tracking system of claim 1 , wherein the adjustment module is configured to utilize an aspect of the at least one object derived from the two dimensional imaging data in order to adjust a corresponding aspect of at least one object represented by the three dimensional object data.

9. The object tracking system of claim 8 , wherein the aspect is number, dimensions and/or position of the at least one object.

10. The object tracking system of claim 1 , wherein the adjustment module is configured to project the first bounding box into the second bounding box and execute the bounding box regression on the projected bounding box to obtain at least one adjusted two-dimensional bounding box and to perform a reverse of the projection on the at least one adjusted bounding box to obtain at least one adjusted three-dimensional bounding box as the adjusted three-dimensional object data.

11. The object tracking system of claim 10 , wherein the first bounding box constitutes a first estimate of dimensions and/or position of the at least one object and the bounding box regression corrects the first estimate based on truer dimensions and/or position data derivable from the two dimensional imaging data than the dimensions and/or position data derivable from the three dimensional imaging data.

12. The object tracking system of claim 1 , wherein the three dimensional imaging data is obtained from a LIDAR imaging device.

13. The object tracking system of claim 1 , wherein the two dimensional imaging data is obtained from a visual camera.

14. The object tracking system of claim 1 , comprising a second two dimensions to three dimensions transformation module configured to transform the two dimensional object data to further three dimensional object data, wherein the tracking module is configured to track at least one further object based on the further three dimensional object data.

15. The object tracking system of claim 14 , wherein the second two dimensions to three dimensions transformation module is configured to receive predetermined height map data, to determine a range of the at least one two dimensional object in three dimensional space based on a projection of the at least one two dimensional object into the height map data and to transform the two dimensional object data to the further three dimensional object data based on the projection and the height map data.

16. A vehicle, comprising:
a sensor system comprising imaging devices for obtaining three dimensional imaging data and two dimensional imaging data;
a processor configured to:
demarcate at least one three dimensional object to obtain three-dimensional object data;
adjust the three dimensional object data based on the two dimensional imaging data and to output adjusted three dimensional object data;
track at least one object using the adjusted three dimensional object data and to responsively output object tracking data; and
an autonomous vehicle control system configured to control at least one vehicle feature based on the object tracking data,
wherein the processor is configured to determine at least one three dimensional bounding box for the at least one three dimensional object as the three-dimensional object data, wherein the processor is configured to project the at least one three dimensional bounding box into two dimensional image space of the two dimensional imaging data to obtain at least one two dimensional bounding box and is configured to refine a size and a dimension of the at least one two dimensional bounding box using bounding box regression based on the two dimensional imaging data, and wherein the processor is configured transform the at least one two dimensional bounding box to at least one three-dimensional bounding box as the three-dimensional object data.

17. A control method, comprising:
receiving three dimensional imaging data;
receiving two dimensional imaging data;
estimating, by a processor, a three dimensional geometry of at least one object based on the three dimensional imaging data;
adjusting, by the processor, the estimated three dimensional geometry of the at least one object based on the two dimensional imaging data;
tracking, by the processor, the at least one object using the adjusted three dimensional geometry of the at least one object for use in autonomous vehicle control,
wherein estimating the three dimensional geometry comprises determining at least one three dimensional bounding box for the at least one object based on the three dimensional imaging data, wherein adjusting the estimated three dimensional geometry comprises projecting the at least one three dimensional bounding box into at least one two dimensional bounding box registered in the two dimensional imaging data and performing bounding box regression based on the at least one object appearing in the two dimensional imaging data and based on the at least one two dimensional bounding box to adjust the geometry of at least one two dimensional bounding box and reversing the projecting to determine the adjusted three dimensional geometry of the at least one object.

Assignees

• Gm Global Tech Operations Llc

Inventors

• Gao Peter

Classifications

• B60W2554/00

  Input parameters relating to objects · CPC title

• B60R11/04

  Mounting of cameras operative during drive; Arrangement of controls thereof relative to the vehicle · CPC title

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title

• G06T2207/30261

  Obstacle · CPC title

• G06T7/37

  using transform domain methods · CPC title