System and method for vehicle position and velocity estimation based on camera and lidar data

Patent metadata

Field	Value
Publication number	US-10552691-B2
Application number	US-201715496113-A
Country	US
Kind code	B2
Filing date	Apr 25, 2017
Priority date	Apr 25, 2017
Publication date	Feb 4, 2020
Grant date	Feb 4, 2020

Abstract

A vehicle position and velocity estimation based on camera and LIDAR data are disclosed. A particular embodiment includes: receiving input object data from a subsystem of an autonomous vehicle, the input object data including image data from an image generating device and distance data from a distance measuring device; determining a two-dimensional (2D) position of a proximate object near the autonomous vehicle using the image data received from the image generating device; tracking a three-dimensional (3D) position of the proximate object using the distance data received from the distance measuring device over a plurality of cycles and generating tracking data; determining a 3D position of the proximate object using the 2D position, the distance data received from the distance measuring device, and the tracking data; determining a velocity of the proximate object using the 3D position and the tracking data; and outputting the 3D position and velocity of the proximate object relative to the autonomous vehicle.

First claim

What is claimed is:

1. A system comprising:
a data processor; and
a vehicle position and velocity estimation module, executable by the data processor, the vehicle position and velocity estimation module being configured to perform a proximate object position and velocity estimation operation for an autonomous vehicle, the proximate object position and velocity estimation operation being configured to:
receive input object data from a subsystem of the autonomous vehicle, the input object data including image data from an image generating device and distance data from a distance measuring device, the distance measuring device being one or more light imaging, detection, and ranging (LIDAR) sensors;
determine a two-dimensional (2D) position of a proximate object near the autonomous vehicle using the image data received from the image generating device and semantic segmentation processing of the image data;
track a three-dimensional (3D) position of the proximate object using the distance data received from the distance measuring device over a plurality of cycles and generate tracking data;
correlate the proximate object identified from the image data with the proximate object identified and tracked from the distance data, the correlation being configured to match the 2D position of the proximate object detected in the image data with the 3D position of the same proximate object detected in the distance data;
determine a 3D position of the proximate object using the 2D position, the distance data received from the distance measuring device, and the tracking data;
determine a velocity of the proximate object using the 3D position and the tracking data; and
output the 3D position and velocity of the proximate object relative to the autonomous vehicle.

2. The system of claim 1 wherein the image generating device is one or more cameras.

3. The system of claim 1 wherein the proximate object position and velocity estimation operation being further configured to retain position tracking data from a previous cycle.

4. The system of claim 1 wherein the image data corresponds to at least one frame from a video stream generated by one or more cameras.

5. The system of claim 1 wherein the distance data corresponds to a point cloud generated by the one or more LIDAR sensors.

6. The system of claim 1 wherein the 3D position and velocity of the proximate object is output to a vehicle control subsystem causing the autonomous vehicle to avoid the proximate object.

7. The system of claim 1 wherein the proximate object position and velocity estimation operation being configured to generate the 3D position and velocity of the proximate object even when intermittent, errant, or unstable image data or distance data is received.

8. A method comprising:
receiving input object data from a subsystem of an autonomous vehicle, the input object data including image data from an image generating device and distance data from a distance measuring device, the distance measuring device being one or more light imaging, detection, and ranging (LIDAR) sensors;
determining a two-dimensional (2D) position of a proximate object near the autonomous vehicle using the image data received from the image generating device and semantic segmentation processing of the image data;
tracking a three-dimensional (3D) position of the proximate object using the distance data received from the distance measuring device over a plurality of cycles and generating tracking data;
correlating the proximate object identified from the image data with the proximate object identified and tracked from the distance data, the correlating including matching the 2D position of the proximate object detected in the image data with the 3D position of the same proximate object detected in the distance data;
determining a 3D position of the proximate object using the 2D position, the distance data received from the distance measuring device, and the tracking data;
determining a velocity of the proximate object using the 3D position and the tracking data; and
outputting the 3D position and velocity of the proximate object relative to the autonomous vehicle.

9. The method of claim 8 wherein the image generating device is one or more cameras.

10. The method of claim 8 including retaining position tracking data from a previous cycle.

11. The method of claim 8 wherein the image data corresponds to at least one frame from a video stream generated by one or more cameras.

12. The method of claim 8 wherein the distance data corresponds to a point cloud generated by the one or more LIDAR sensors.

13. The method of claim 8 wherein the 3D position and velocity of the proximate object is output to a vehicle control subsystem causing the autonomous vehicle to avoid the proximate object.

14. The method of claim 8 including generating the 3D position and velocity of the proximate object even when intermittent, errant, or unstable image data or distance data is received.

15. A non-transitory machine-useable storage medium embodying instructions which, when executed by a machine, cause the machine to:
receive input object data from a subsystem of an autonomous vehicle, the input object data including image data from an image generating device and distance data from a distance measuring device, the distance measuring device being one or more light imaging, detection, and ranging (LIDAR) sensors;
determine a two-dimensional (2D) position of a proximate object near the autonomous vehicle using the image data received from the image generating device and semantic segmentation processing of the image data;
track a three-dimensional (3D) position of the proximate object using the distance data received from the distance measuring device over a plurality of cycles and generate tracking data;
correlate the proximate object identified from the image data with the proximate object identified and tracked from the distance data, the correlation being configured to match the 2D position of the proximate object detected in the image data with the 3D position of the same proximate object detected in the distance data;
determine a 3D position of the proximate object using the 2D position, the distance data received from the distance measuring device, and the tracking data;
determine a velocity of the proximate object using the 3D position and the tracking data; and
output the 3D position and velocity of the proximate object relative to the autonomous vehicle.

16. The non-transitory machine-useable storage medium of claim 15 wherein the proximate object position and velocity estimation operation being configured to generate the 3D position and velocity of the proximate object even when intermittent, errant, or unstable image data or distance data is received.

17. The non-transitory machine-useable storage medium of claim 15 wherein the 3D position and velocity of the proximate object is output to a vehicle control subsystem causing the autonomous vehicle to avoid the proximate object.

Assignees

• TuSimple

Inventors

• Li Chenyang
• Hou Xiaodi
• Liu Siyuan

Classifications

• G01S17/08

  for measuring distance only (indirect measurement G01S17/46; active triangulation systems G01S17/48) · CPC title

• G08G1/166 Primary

  for active traffic, e.g. moving vehicles, pedestrians, bikes · CPC title

• G01S17/86

  Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders · CPC title

• G01S17/931

  of land vehicles · CPC title

• G01S17/88

  Lidar systems specially adapted for specific applications · CPC title