System and method for generating information on remainder of measurement using sensor data

Patent metadata

Field	Value
Publication number	US-12091043-B2
Application number	US-202318109113-A
Country	US
Kind code	B2
Filing date	Feb 13, 2023
Priority date	May 19, 2022
Publication date	Sep 17, 2024
Grant date	Sep 17, 2024

Abstract

A method may include obtaining lidar data comprising a plurality of lidar returns from an environment of an autonomous vehicle. The lidar data may be processed with a machine learning model to generate, for the plurality of lidar returns, a plurality of first outputs that each identify a respective lidar return as belonging to an object or non-object and a plurality of second outputs that identify lidar returns belonging to objects as harmful or non-harmful to the autonomous vehicle. A subset of the lidar returns identified as belonging to objects that (i) do not correspond to any of a plurality of pre-classified objects and (ii) were identified as harmful to the autonomous vehicle may be determined. The autonomous vehicle may be controlled based at least in part on the subset of lidar returns.

First claim

What is claimed is:

1. A method, comprising:
(a) obtaining, by one or more processors from one or more sensors, lidar data comprising a plurality of lidar returns from an environment of an autonomous vehicle;
(b) processing, by the one or more processors, the lidar data with a machine learning model to generate, for the plurality of lidar returns, a plurality of first outputs that each identify a respective lidar return as belonging to an object or non-object and a plurality of second outputs that identify lidar returns belonging to objects as harmful or non-harmful to the autonomous vehicle;
(c) determining, by the one or more processors, that a subset of the lidar returns identified as belonging to objects do not correspond to any of a plurality of pre-classified objects identified by a main detector, by de-duplicating objects identified by the machine learning model and the plurality of pre-classified objects;
(c1) determining, by the one or more processors, that at least one of the objects determined in (c) is harmful to the autonomous vehicle; and
(d) controlling, by the one or more processors, the autonomous vehicle based at least in part on the subset of lidar returns determined in (c).

2. The method of claim 1 , wherein (c) comprises:
(e) identifying an object as harmful or non-harmful to the autonomous vehicle based on a size of a lidar return identified as belonging to the object.

3. The method of claim 2 , wherein (e) comprises:
(f) determining whether the lidar return identified as belonging to the object has the size such that the autonomous vehicle can straddle over the object; and
(g) in response to determining that the lidar return has the size such that the autonomous vehicle cannot straddle over the object, identifying the object as harmful.

4. The method of claim 1 , wherein
the subset of the lidar returns comprises one or more lidar returns corresponding to a first object in a first range ahead of the autonomous vehicle, and
the vehicle is controlled to stop for the first object.

5. The method of claim 1 , wherein
the subset of the lidar returns comprises one or more lidar returns corresponding to a first object in a first range ahead of the autonomous vehicle, and
the vehicle is controlled to make a lane change to avoid the first object.

6. The method of claim 1 , wherein:
(b) comprises processing a set of features relating to the lidar data with the machine learning model to generate a plurality of third outputs that each indicate a property of an object, and
(c) comprises comparing the property of the object to corresponding properties of the plurality of pre-classified objects.

7. The method of claim 6 , further comprising:
prior to (c), aggregating the plurality of first outputs and the plurality of third outputs into a plurality of three dimensional (3-D) voxels.

8. The method of claim 7 , wherein the aggregating comprises averaging subsets of the plurality of first outputs and the plurality of third outputs.

9. The method of claim 6 , wherein the property of the object includes at least one of predicted velocity information or predicted location information associated with the respective lidar return.

10. A system comprising one or more processors and one or more memories operably coupled with the one or more processors, wherein the one or more memories store instructions that, in response to the execution of the instructions by one or more processors, cause the one or more processors to perform the following operations:
(a) obtaining, from one or more sensors, lidar data comprising a plurality of lidar returns from an environment of an autonomous vehicle;
(b) processing the lidar data with a machine learning model to generate, for the plurality of lidar returns, a plurality of first outputs that each identify a respective lidar return as belonging to an object or non-object and a plurality of second outputs that identify lidar returns belonging to objects as harmful or non-harmful to the autonomous vehicle;
(c) determining that a subset of the lidar returns identified as belonging to objects do not correspond to any of a plurality of pre-classified objects identified by a main detector, by de-duplicating objects identified by the machine learning model and the plurality of pre-classified objects;
(c1) determining that at least one of the objects determined in (c) is harmful to the autonomous vehicle; and
(d) controlling the autonomous vehicle based at least in part on the subset of lidar returns determined in (c).

11. The system of claim 10 , wherein in performing (c), the one or more processors are configured to perform:
(e) identifying an object as harmful or non-harmful to the autonomous vehicle based on a size of a lidar return identified as belonging to the object.

12. The system of claim 11 , wherein in performing (e), the one or more processors are configured to perform:
(f) determining whether the lidar return identified as belonging to the object has the size such that the autonomous vehicle can straddle over the object; and
(g) in response to determining that the lidar return has the size such that the autonomous vehicle cannot straddle over the object, identifying the object as harmful.

13. The system of claim 10 , wherein
the subset of the lidar returns comprises one or more lidar returns corresponding to a first object in a first range ahead of the autonomous vehicle, and
the one or more processors are configured to control the vehicle to stop for the first object.

14. The system of claim 10 , wherein
the subset of the lidar returns comprises one or more lidar returns corresponding to a first object in a first range ahead of the autonomous vehicle, and
the one or more processors are configured to control the vehicle to make a lane change to avoid the first object.

15. The system of claim system 10 , wherein:
in performing (b), the one or more processors are configured to process a set of features relating to the lidar data with the machine learning model to generate a plurality of fourth outputs that each indicate a property of an object, and
in performing (c), the one or more processors are configured to compare the property of the object to corresponding properties of the plurality of pre-classified objects.

16. The system of claim 15 , wherein prior to performing (c), the one or more processors are configured to aggregate the plurality of first outputs and the plurality of fourth outputs into a plurality of three dimensional (3-D) voxels.

17. The system of claim 16 , wherein in performing the aggregating, the one or more processors are configured to average subsets of the plurality of first outputs and the plurality of fourth outputs.

18. The system of claim 15 , wherein the property of the object includes at least one of predicted velocity information or predicted location information associated with the respective lidar return.

Assignees

• Aurora Operations Inc

Inventors

• Charland Jake
• Eade Ethan
• Lakshmanan Karthik
• Munoz Daniel
• Sean Samuel
• Xie Yuchen
• Yang Luona

Classifications

• G06V10/811

  the classifiers operating on different input data, e.g. multi-modal recognition · CPC title

• G06V20/58

  Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads · CPC title

• B60W2420/408

  Radar; Laser, e.g. lidar · CPC title

• B60W2420/403

  Image sensing, e.g. optical camera · CPC title

• B60W2510/0638

  Engine speed · CPC title