Automatic annotation of object trajectories in multiple dimensions

Patent metadata

Field	Value
Publication number	US-12116015-B2
Application number	US-202117528559-A
Country	US
Kind code	B2
Filing date	Nov 17, 2021
Priority date	Nov 17, 2020
Publication date	Oct 15, 2024
Grant date	Oct 15, 2024

Abstract

Techniques for improving the performance of an autonomous vehicle (AV) by automatically annotating objects surrounding the AV are described herein. A system can obtain sensor data from a sensor coupled to the AV and generate an initial object trajectory for an object using the sensor data. Additionally, the system can determine a fixed value for the object size of the object based on the initial object trajectory. Moreover, the system can generate an updated initial object trajectory, wherein the object size corresponds to the fixed value. Furthermore, the system can determine, based on the sensor data and the updated initial object trajectory, a refined object trajectory. Subsequently, the system can generate a multi-dimensional label for the object based on the refined object trajectory. A motion plan for controlling the AV can be generated based on the multi-dimensional label.

First claim

What is claimed is:

1. A method comprising:
(a) obtaining, from one or more sensors attached to an autonomous vehicle (AV), sensor data, the sensor data being sequential over a period of time;
(b) generating, using the sensor data, an initial object trajectory for an object, the initial object trajectory comprising a plurality of initial object observations respectively having an object size of the object, an initial object pose of the object, and a timestamp;
(c) determining, based on the sensor data and the plurality of initial object observations, a fixed value for the object size of the object, wherein (c) comprises providing the plurality of initial object observations to an object encoder and an object decoder, wherein the object encoder extracts high-resolution features of the object in a bird's-eye-view (BEV) space in order to determine the fixed value;
(d) generating an updated initial object trajectory comprising a plurality of updated initial object observations, wherein the object size in the plurality of updated initial object observations corresponds to the fixed value;
(e) determining, based on the sensor data and the updated initial object trajectory, a refined object trajectory comprising a plurality of refined object observations respectively comprising an updated object pose of the object for the plurality of refined object observations;
(f) generating a multi-dimensional label for the object based on the refined object trajectory; and
(g) generating a motion plan for the AV based on the multi-dimensional label, wherein the AV is controlled using the motion plan.

2. The method of claim 1 , wherein the multi-dimensional label for the object comprises a bounding box defined relative to a plurality of spatial dimensions and a time dimension corresponding to the period of time.

3. The method of claim 1 , wherein the initial object pose comprises a center position of the object at the timestamp and an orientation of the object at the timestamp.

4. The method of claim 3 , wherein (c) comprises converting the sensor data into an object coordinate system by aligning the center position of the object and the orientation of the object over multiple timestamps in the plurality of initial object observations.

5. The method of claim 1 , wherein (e) comprises converting the sensor data into a world coordinate system, wherein a motion of the object is determined independent of a movement associated with the AV.

6. The method of claim 1 , wherein (e) comprises providing the plurality of updated initial object observations to a path encoder and a path decoder, and wherein the path encoder extracts spatial-temporal features from four-dimensional point clouds that are generated from the sensor data.

7. The method of claim 1 , wherein (e) comprises determining a height dimension of the object based on ground data obtained from high-definition maps, and wherein the multi-dimensional label for the object includes the height dimension of the object.

8. The method of claim 1 , wherein the one or more sensors comprises a LiDAR sensor, and wherein the sensor data comprises a series of sequential point clouds of LiDAR data.

9. The method of claim 1 , wherein the object is one of a vehicle, a pedestrian, or a bicycle.

10. The method of claim 1 , further comprising:
training a 4D label generation model using the multi-dimensional label for the object.

11. The method of claim 10 , further comprising:
detecting a second object around the AV using the sensor data; and
determining, using the trained 4D label generation model, a multi-dimensional label for the second object.

12. A computing system comprising:
one or more processors; and
one or more computer-readable media storing instructions for execution by the one or more processors to cause the computing system to perform operations, the operations comprising:
(a) obtaining, from one or more sensors attached to an autonomous vehicle (AV), sensor data, the sensor data being sequential over a period of time;
(b) generating, using the sensor data, an initial object trajectory for an object, the initial object trajectory comprising a plurality of initial object observations respectively having an object size of the object, an initial object pose of the object, and a timestamp;
(c) determining, based on the sensor data and the plurality of initial object observations, a fixed value for the object size of the object, wherein (c) comprises providing the plurality of initial object observations to an object encoder and an object decoder, wherein the object encoder extracts high-resolution features of the object in a bird's-eye-view (BEV) space in order to determine the fixed value;
(d) generating an updated initial object trajectory comprising a plurality of updated initial object observations, wherein the object size in the plurality of updated initial object observations corresponds to the fixed value;
(e) determining, based on the sensor data and the updated initial object trajectory, a refined object trajectory comprising a plurality of refined object observations respectively comprising an updated object pose of the object for the plurality of refined object observations;
(f) generating a multi-dimensional label for the object based on the refined object trajectory; and
(g) generating a motion plan for the AV based on the multi-dimensional label, wherein the AV is controlled using the motion plan.

13. The computing system of claim 12 , the operations further comprising:
training a 4D label generation model using the multi-dimensional label for the object;
detecting a second object around the AV using the sensor data; and
determining, using the trained 4D label generation model, a motion path for the second object.

14. The computing system of claim 12 , wherein (e) comprises providing the plurality of updated initial object observations to a path encoder and a path decoder, and wherein the path encoder extracts spatial-temporal features from four-dimensional point clouds that are generated from the sensor data.

15. An autonomous vehicle (AV) control system of an AV comprising:
one or more machine-learned models, wherein the one or more machine-learned models have been learned via performance of machine learning algorithms on one or more training examples comprising motion path data, the motion path data having been generated by performance of operations, the operations comprising:
(a) obtaining, from one or more sensors attached to an autonomous vehicle (AV), sensor data, the sensor data being sequential over a period of time;
(b) generating, using the sensor data, an initial object trajectory for an object, the initial object trajectory comprising a plurality of initial object observations respectively having an object size of the object, an initial object pose of the object, and a timestamp;
(c) determining, based on the sensor data and the plurality of initial object observations, a fixed value for the object size of the object, wherein (c) comprises providing the plurality of initial object observations to an object encoder and an object decoder, wherein the object encoder extracts high-resolution features of the object in a bird's-eye-view (BEV) space in order to determine the fixed value;
(d) generating an updated initial object trajectory comprising a plurality of updated initial object observations, wherein the object size in the plurality of updated initial object observations corresponds to the fixed value;
(e) determining, based on the sensor data and the updated initial object trajectory, a refined object trajectory comprising a plurality of

Assignees

• Aurora Operations Inc

Inventors

• Yang Bin
• Liang Ming
• Zeng Wenyuan
• Bai Min
• Urtasun Raquel

Classifications

• G06N3/0455

  Auto-encoder networks; Encoder-decoder networks · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/08

  Learning methods · CPC title

• G05D1/249

  from positioning sensors located off-board the vehicle, e.g. from cameras · CPC title