Autonomous driving LiDAR technology

Patent metadata

Field	Value
Publication number	US-12466433-B2
Application number	US-202218070102-A
Country	US
Kind code	B2
Filing date	Nov 28, 2022
Priority date	Dec 15, 2021
Publication date	Nov 11, 2025
Grant date	Nov 11, 2025

Abstract

Autonomous vehicles can include systems and apparatus for performing signal processing on point cloud data from Light Detection and Ranging (LiDAR) devices located on the autonomous vehicles. A method includes obtaining, by a computer located in an autonomous vehicle, a combined point cloud data that describes a plurality of areas of an environment in which the autonomous vehicle is operating; determining that a first set of points from the combined point cloud data are located within fields of view of cameras located on the autonomous vehicle; assigning one or more labels to a second set of points from the first set of points in response to determining that the second set of points are located within bounding box(es) around object(s) in images obtained from the cameras; and causing the autonomous vehicle to operate based on characteristic(s) of the object(s) determined from the second set of points.

First claim

What is claimed is:

1 . A method of autonomous vehicle operation, comprising:
obtaining, by a computer located in an autonomous vehicle, a combined point cloud data that describes a plurality of areas of an environment in which the autonomous vehicle is operating,
wherein the combined point cloud data is obtained by performing a signal processing technique on multiple sets of point cloud data obtained from a plurality of light detection and ranging sensors located on the autonomous vehicle;
wherein the combined point cloud data comprises a first set of combined point cloud data combined with a second set of combined point cloud data, wherein the second set of combined point cloud data is obtained or scanned later than the first set of combined point cloud data, wherein the first set of combined point cloud data is obtained by combining a first set of point cloud data of at least two light detection and ranging sensors of the plurality of light detection and ranging sensors;
determining, from the combined point cloud data, a first set of points located within a plurality of fields of view of a plurality of cameras located on the autonomous vehicle; determining, from the first set of points, a second set of points located within one or more bounding boxes around one or more objects in images obtained from the plurality of cameras, assigning one or more labels to the second set of points, wherein the one or more labels include information that identifies the one or more objects; causing the autonomous vehicle to operate based on one or more characteristics of the one or more objects determined from the second set of points; enlarging the bounding boxes by a deep fuse encoder to include more contextual points; adding, by the deep fuse encoder, virtual points to each bounding box as size-aware point features; and extracting and normalizing the points within the enlarged boxes in a canonical coordinate system to form input data of a deep fusion encoder; wherein the deep fusion encoder comprises a Multi-Layer Perceptron (MLP) module with a plurality of fully connected layers and a max-pooling operator for feature aggregation.

2 . The method of claim 1 , wherein the signal processing technique to obtain the combined point cloud data comprises:
receiving, from each of at least two light detection and ranging sensors of the plurality of light detection and ranging sensors, the first set of point cloud data of at least two areas of the plurality of areas of the environment, wherein the multiple sets of point cloud data include the first set of point cloud data that is scanned or obtained at a first time; and obtaining the first set of combined point cloud data by combining the first set of point cloud data of each of the least two light detection and ranging sensors.

3 . The method of claim 2 , further comprising:
receiving, from each of the at least two light detection and ranging sensors, a second set of point cloud data of at least some of the at least two areas, wherein the multiple sets of point cloud data include the second set of point cloud data that is scanned or obtained at a second time later than the first time; obtaining the second set of combined point cloud data by combining the second set of point cloud data of each of the least two light detection and ranging sensor.

4 . The method of claim 3 , wherein the first set of combined point cloud data and the second set of combined point cloud data are obtained by:
projecting the first set of point cloud data of each of the at least two light detection and ranging sensors onto a three dimensional inertial measurement unit coordinate system using first extrinsic parameters,
wherein the first extrinsic parameters describe a spatial relationship between each of the at least two light detection and ranging sensors and an inertial measurement unit located on or in the autonomous vehicle; and
projecting the second set of point cloud data of each of the at least two light detection and ranging sensors onto the three dimensional inertial measurement unit coordinate system using the first extrinsic parameters.

5 . The method of claim 4 , wherein the first extrinsic parameters include sets of parameters that are unique to each of the at least two light detection and ranging sensors.

6 . The method of claim 3 , wherein obtaining the combined point cloud data comprises: obtaining a plurality of sets of combined point cloud data, each set includes the first set of combined point cloud data and the second set of combined point cloud data, and wherein a number of the plurality of sets of combined point cloud data is predetermined.

7 . The method of claim 3 , wherein each set of point cloud data from the multiple sets of point cloud data is scanned or obtained by a light detection and ranging sensor within a time window, and wherein the first time and the second time are within the time window.

8 . The method of claim 3 , wherein the first set of combined point cloud data is combined with the second set of combined point cloud data by:
obtaining a transformed set of point cloud data by transforming the first set of combined point cloud data and the second set of combined point cloud data to a global coordinate system; and obtaining the combined point cloud data by transforming the transformed set of point cloud data to three dimensional inertial measurement unit coordinates associated with the second time when the second set of point cloud data was obtained or scanned by the least two light detection and ranging sensors.

9 . A system for autonomous vehicle operation, the system comprising a computer that comprises:
at least one processor and at least one memory including computer program code which, when executed by the at least one processor, cause the computer to at least: obtain a combined point cloud data that describes a plurality of areas of an environment in which an autonomous vehicle is operating,
wherein the combined point cloud data is obtained by performing a signal processing technique on multiple sets of point cloud data obtained from a plurality of light detection and ranging sensors located on the autonomous vehicle,
wherein the combined point cloud data comprises a first set of combined point cloud data combined with a second set of combined point cloud data, wherein the second set of combined point cloud data is obtained or scanned later than the first set of combined point cloud data, wherein the first set of combined point cloud data is obtained by combining a first set of point cloud data of at least two light detection and ranging sensors of the plurality of light detection and ranging sensors;
wherein the computer is located in the autonomous vehicle;
determine, from the combined point cloud data, a first set of points located within a plurality of fields of view of a plurality of cameras located on the autonomous vehicle; determine, from the first set of points, a second set of points located within one or more bounding boxes around one or more objects in images obtained from the plurality of cameras, assign one or more labels to the second set of points, wherein the one or more labels include information that identifies the one or more objects; cause the autonomous vehicle to operate based on one or more characteristics of the one or more objects determined from the second set of points; enlarge the bounding boxes by a deep fuse encoder to include more contextual points; add, by the deep fuse encoder, virtual points to each bounding box as size-aware point features; and extract and normalize the points within the enlarged boxes in a canonical coordinate system to form input data of a deep fusion encoder; wherein the deep fu

Assignees

• Tusimple Inc

Inventors

• Wang Panqu
• Wang Yu
• Zhao Xiangchen
• Ye Dongqiangzi

Classifications

• B60W2420/408

  Radar; Laser, e.g. lidar · CPC title

• B60W2420/403

  Image sensing, e.g. optical camera · CPC title

• G01S17/89

  for mapping or imaging · CPC title

• B60W2300/145

  Semi-trailers · CPC title

• G01S17/931

  of land vehicles · CPC title