System and method for vehicle position and velocity estimation based on camera and LIDAR data

Patent metadata

Field	Value
Publication number	US-11557128-B2
Application number	US-202016752637-A
Country	US
Kind code	B2
Filing date	Jan 25, 2020
Priority date	Apr 25, 2017
Publication date	Jan 17, 2023
Grant date	Jan 17, 2023

Abstract

A vehicle position and velocity estimation based on camera and LIDAR data are disclosed. A particular embodiment includes: receiving input object data from a subsystem of an autonomous vehicle, the input object data including image data from an image generating device and distance data from a distance measuring device; determining a two-dimensional (2D) position of a proximate object near the autonomous vehicle using the image data received from the image generating device; tracking a three-dimensional (3D) position of the proximate object using the distance data received from the distance measuring device over a plurality of cycles and generating tracking data; determining a 3D position of the proximate object using the 2D position, the distance data received from the distance measuring device, and the tracking data; determining a velocity of the proximate object using the 3D position and the tracking data; and outputting the 3D position and velocity of the proximate object relative to the autonomous vehicle.

First claim

What is claimed is:

1. A system comprising:
a data processor; and
a vehicle position and velocity estimation module which, when executed by the data processor, causes the system to:
receive input object data from a subsystem of an autonomous vehicle, the input object data including image data from an image generating device and distance data from a distance measuring device, the distance measuring device comprising one or more light imaging, detection, and ranging (LIDAR) sensors;
determine a first position of a proximate object near the autonomous vehicle from the image data, the first position being determined using semantic segmentation processing of the image data, the semantic segmentation processing including assigning an object categorical label to each pixel in the image data;
determine a second position of the proximate object from the distance data, the second position being tracked using the distance data received from the distance measuring device over a plurality of processing cycles, the distance data being collected during at least one of the plurality of processing cycles;
correlate the first position and the second position by matching the first position of the proximate object detected in the image data with the second position of the same proximate object detected in the distance data;
determine a three-dimensional (3D) position of the proximate object using the correlated first and second positions; and
track the 3D position of the proximate object over a plurality of processing cycles, the input object data being collected during at least one of the plurality of processing cycles.

2. The system of claim 1 wherein the image generating device is one or more cameras- and the second position is a 3D position.

3. The system of claim 1 being further configured to determine the first position as a two-dimensional (2D) position of the proximate object using the image data received from the image generating device.

4. The system of claim 1 being further configured to determine the 3D position of the proximate object using a two-dimensional (2D) position and a point cloud of the distance data received from the distance measuring device.

5. The system of claim 1 being further configured to retain the 3D position of the proximate object as tracking data over the plurality of processing cycles.

6. The system of claim 5 being further configured to determine a velocity of the proximate object using the 3D position and the tracking data.

7. The system of claim 1 wherein the second position is a 3D position and the system is further configured to track the second position using the distance data received from the distance measuring device over a plurality of processing cycles, the distance data being collected during at least one of the plurality of processing cycles.

8. The system of claim 1 being further configured to output the 3D position of the proximate object to a trajectory planning module of the autonomous vehicle.

9. A method comprising:
receiving input object data from a subsystem of an autonomous vehicle, the input object data including image data from an image generating device and distance data from a distance measuring device, the distance measuring device comprising one or more light imaging, detection, and ranging (LIDAR) sensors;
determining a first position of a proximate object near the autonomous vehicle from the image data, the first position being determined using semantic segmentation processing of the image data, the semantic segmentation processing including assigning an object categorical label to each pixel in the image data;
determining a second position of the proximate object from the distance data, the second position being tracked using the distance data received from the distance measuring device over a plurality of processing cycles, the distance data being collected during at least one of the plurality of processing cycles;
correlating the first position and the second position by matching the first position of the proximate object detected in the image data with the second position of the same proximate object detected in the distance data;
determining a three-dimensional (3D) position of the proximate object using the correlated first and second positions; and
tracking the 3D position of the proximate object over a plurality of processing cycles, the input object data being collected during at least one of the plurality of processing cycles.

10. The method of claim 9 wherein the image generating device is one or more cameras.

11. The method of claim 9 wherein the second position is a 3D position.

12. The method of claim 9 including determining the first position as a two-dimensional (2D) position of the proximate object using the image data received from the image generating device, the image generating device being a plurality of cameras.

13. The method of claim 9 including determining the 3D position of the proximate object using a two-dimensional (2D) position and a point cloud generated by one or more LIDAR sensors.

14. The method of claim 9 including retaining the 3D position of the proximate object as tracking data over the plurality of processing cycles in a memory device of an in-vehicle control system.

15. The method of claim 9 including tracking the 3D position of the proximate object using the distance data received from the distance measuring device over a plurality of cycles and determining a velocity of the proximate object using the 3D position and the tracking data.

16. The method of claim 9 including correlating positions and velocities of proximate objects found in a previous processing cycle with proximate objects identified in a current processing cycle.

17. The method of claim 9 including generating an alert if the 3D position of the proximate object may intersect with a trajectory of the autonomous vehicle.

18. A non-transitory machine-useable storage medium embodying instructions which, when executed by at least one processor, cause the at least one processor to:
receive input object data from a subsystem of an autonomous vehicle, the input object data including image data from an image generating device and distance data from a distance measuring device, the distance measuring device comprising one or more light imaging, detection, and ranging (LIDAR) sensors;
determine a first position of a proximate object near the autonomous vehicle from the image data, the first position being determined using semantic segmentation processing of the image data, the semantic segmentation processing including assigning an object categorical label to each pixel in the image data;
determine a second position of the proximate object from the distance data, the second position being tracked using the distance data received from the distance measuring device over a plurality of processing cycles, the distance data being collected during at least one of the plurality of processing cycles;
correlate the first position and the second position by matching the first position of the proximate object detected in the image data with the second position of the same proximate object detected in the distance data;
determine a three-dimensional (3D) position of the proximate object using the correlated first and second positions; and
track the 3D position of the proximate object over a plurality of processing cycles, the input object data being collected during at least one of the plurality of processing cycles.

19. The non-transitory machine-useable storage medium of claim

Assignees

• Tusimple Inc

Inventors

• Li Chenyang
• Hou Xiaodi
• Liu Siyuan

Classifications

• G01S17/931

  of land vehicles · CPC title

• G01S17/66

  Tracking systems using electromagnetic waves other than radio waves · CPC title

• G08G1/166 Primary

  for active traffic, e.g. moving vehicles, pedestrians, bikes · CPC title

• G01S7/4808 Primary

  Evaluating distance, position or velocity data · CPC title

• G06V20/58 Primary

  Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads · CPC title