Object tracking

Patent metadata

Field	Value
Publication number	US-10318822-B2
Application number	US-201715480437-A
Country	US
Kind code	B2
Filing date	Apr 6, 2017
Priority date	Apr 6, 2017
Publication date	Jun 11, 2019
Grant date	Jun 11, 2019

Abstract

Methods and systems are provided for tracking an object. The system includes a data receiving module receiving two dimensional imaging data including an object and height map data correlating ground height and location. A two dimensions to three dimensions transformation module determines a location of the object in three dimensional space based on the two dimensional imaging data and the height map data. A tracking module tracks the object using the location of the object.

First claim

What is claimed is:

1. An object tracking system for a vehicle having one or more vehicle control components, comprising:
a vehicle control module in communication with the one or more control components, the vehicle control module including:
a non-transitory computer readable medium comprising:
a data storage module configured to, by a processor, store in a data storage device height map data correlating ground height and location and generated from lidar data;
a data receiving module configured to, by a processor, receive two dimensional imaging data generated by a camera and including at least one object;
a two dimensions to three dimensions transformation module configured to, by a processor, determine a location of the camera in the height map data based on a real world location of the camera, project the at least one object of the two dimensional imaging data into the height map data based on the location and ray tracing to determine a distance of the object from the camera, and determine a location of the at least one object in a three dimensional space based on the distance;
a tracking module configured to, by a processor, track at least one object using the location of the at least one object in the three dimensional space and generate tracking data based thereon; and
the vehicle control module being configured to actuate at least one of the one or more control components based, in part, on the tracking data.

2. The object tracking system of claim 1 , wherein the two dimensions to three dimensions transformation module is further configured to project the at least one object of the two dimensional imaging data into the height map data to determine a ground intersection of the at least one object and, based on the ground intersection and the correlation of ground height and location in the height map data to determine the distance of the object from the camera.

3. The object tracking system of claim 1 , wherein the data storage module is configured to store calibration data including camera pose data, and wherein the two dimensions to three dimensions transformation module is configured to project the at least one object of the two dimensional imaging data into the height map data from the camera pose.

4. The object tracking system of claim 1 , comprising an object identification module configured to demarcate the at least one object in the two dimensional imaging data to obtain two dimensional object data, and wherein the two dimensions to three dimensions transformation module is configured to determine the location of the at least one object in the three dimensional space based on the two dimensional object data.

5. The object tracking system of claim 4 , wherein the object identification module is configured to determine at least one bounding box as the two dimensional object data.

6. The object tracking system of claim 4 , wherein the two dimensions to three dimensions transformation module is configured to transform a bottom of the at least one bounding box to a ground intersection in the height map data in determining the location of the at least one object in three dimensional space.

7. The object tracking system of claim 1 , comprising a visual classification module configured to run a neural network to classify the at least one object and to determine dimensions of the at least one object based on the classification, wherein the tracking module is configured to track at least one object using the location of the at least one object and the dimensions of the at least one object.

8. The object tracking system of claim 7 , comprising an object identification module configured to demarcate the at least one object to obtain at least one bounding box, wherein the visual classification module is configured to perform bounding box regression on the at least one bounding box using the neural network to obtain at least one regressed bounding box, and wherein the two dimensions to three dimensions transformation module is configured to determine the location of the at least one object in three dimensional space based on the two dimensional imaging data, the height map data and the at least one regressed bounding box.

9. The object tracking system of claim 1 , wherein the data receiving module is configured to receive three dimensional imaging data including at least one other object, wherein the tracking module is configured to track the at least one object based on the two dimensional imaging data and the at least one other object based on the three dimensional imaging data.

10. An autonomous vehicle, comprising:
a sensor system comprising at least one camera that generates two dimensional imaging data including at least one object;
a data storage device configured to store height map data correlating ground height and location;
a processor configured to:
determine a location of the at least one camera in the height map data based on a real world location of the autonomous vehicle and a pose of the at least one camera,
project the at least one object of the two dimensional imaging data into the height map data based on the location and ray tracing to determine a distance of the object from the camera,
determine a location of the at least one object in a three dimensional space based on the distance;
track at least one object using the location of the at least one object and to responsively output tracking data; and
an autonomous vehicle control module configured to control one or more vehicle components based, in part, on the tracking data.

11. The autonomous vehicle of claim 10 , wherein the processor is configured to transform a height of the at least one object in the two dimensional imaging data to a ground intersection in the height map data and determine the distance based on the ground intersection.

12. The autonomous vehicle of claim 10 , wherein the processor is configured to demarcate the at least one object to obtain at least one bounding box and to perform bounding box regression on the at least one bounding box using a neural network to obtain at least one regressed bounding box, and wherein processor is configured to determine the location of the at least one object in the three dimensional space based on the at least one regressed bounding box.

13. The autonomous vehicle of claim 12 , wherein the processor is configured to transform a bottom of the at least one bounding box to a ground intersection in the height map data in determining the location of the at least one object in three dimensional space.

14. A method of tracking at least one object, comprising:
storing, via a processor, height map data correlating ground height and real world location;
receiving, via a processor, two dimensional imaging data including at least one object;
determining, via a processor, a location of the camera in the height map data based on a real world location of the camera;
projecting, via a processor, the at least one object of the two dimensional imaging data into the height map data based on the location and ray tracing to determine a distance of the object from the camera;
determining, via a processor, a location of the at least one object in a three dimensional space based on the distance;
tracking, via a processor, at least one object using the location of the at least one object for use in autonomous vehicle control; and
exercising vehicle control based, in part, on the tracking.

15. The method of claim 14 , comprising receiving, via a processor, calibration data including camera pose data spatially relating the autonomous vehicle and the at least one camera, determining, via a processor,

Assignees

• Gm Global Tech Operations Llc

Inventors

• Gao Peter
• Minster Gautier
• Branson Elliot
• Glaser Stuart

Classifications

• G06V20/56

  exterior to a vehicle by using sensors mounted on the vehicle · CPC title

• G08G1/202 Primary

  Dispatching vehicles on the basis of a location, e.g. taxi dispatching · CPC title

• B60W2554/00

  Input parameters relating to objects · CPC title

• G06T2207/30252

  Vehicle exterior; Vicinity of vehicle · CPC title

• B60W10/04

  including control of propulsion units · CPC title