Object-centric three-dimensional auto labeling of point cloud data

Patent metadata

Field	Value
Publication number	US-12073575-B2
Application number	US-202117407795-A
Country	US
Kind code	B2
Filing date	Aug 20, 2021
Priority date	Aug 21, 2020
Publication date	Aug 27, 2024
Grant date	Aug 27, 2024

Abstract

Methods, computer systems, and apparatus, including computer programs encoded on computer storage media, for performing three-dimensional auto-labeling on sensor data. The system obtains a sensor data segment that includes a temporal sequence of three-dimensional point clouds generated from sensor readings of an environment by one or more sensors. The system identifies, from the sensor data segment, (i) a plurality of object tracks that each corresponds to a different object in the environment and (ii) for each object track, respective initial three-dimensional regions in each of one or more of the point clouds in which the corresponding object appears. The system generates, for each object track, extracted object track data that includes at least the points in the respective initial three-dimensional regions for the object track. The system further generates, for each object track and from the extracted object track data for the object track, an auto labeling output that defines respective refined three-dimensional regions in each of the one or more point clouds.

First claim

What is claimed is:

1. A method comprising:
obtaining a sensor data segment comprising a temporal sequence of three-dimensional point clouds generated from sensor readings of an environment by one or more sensors, each three-dimensional point cloud comprising a respective plurality of points in a first coordinate system;
identifying, from the sensor data segment, (i) a plurality of object tracks that each corresponds to a different object in the environment and (ii) for each object track, respective initial three-dimensional regions in each of one or more of the point clouds in which the corresponding object appears, wherein each initial three-dimensional region is an initial estimate of the three-dimensional region of the point cloud that includes points that are measurements of the corresponding object, wherein the identifying comprises:
processing each of the point clouds in the temporal sequence using an object detector to obtain, for each point cloud, a detector output that identifies a plurality of three-dimensional regions in the point cloud that are predicted to correspond to objects; and
processing the detector output using an object tracker to obtain an object tracker output that associates each of at least a subset of the three-dimensional regions in each of the point clouds with a respective one of the plurality of object tracks;

generating, for each object track, extracted object track data that includes at least the points in the respective initial three-dimensional regions for the object track; and
generating, for each object track and from the extracted object track data for the object track, an auto labeling output that defines respective refined three-dimensional regions in each of the one or more point clouds that is a refined estimate of the three-dimensional region of the point cloud that includes points that are measurements of the corresponding object.

2. The method of claim 1 , wherein the one or more sensors are located on an object in the environment and wherein the first coordinate system is centered at the object in the environment.

3. The method of claim 2 , wherein the object in the environment is an autonomous vehicle navigating through the environment.

4. The method of claim 2 , wherein generating, for each object track, extracted object track data that includes at least the points in the initial three-dimensional regions for the object track comprises:
extracting, from each of the one or more point clouds in which the corresponding object appears, a plurality of points including the points in the initial three-dimensional region in the point cloud for the object track; and
transforming, using frame pose data for each of the one or more point clouds, each of the extracted points to a second coordinate system centered at a stationary location in the environment.

5. The method of claim 4 , wherein the plurality of points includes the points in the initial three-dimensional region of the point cloud for the object track and additional context points in a vicinity of the three-dimensional region in the point cloud.

6. The method of claim 4 , wherein generating, for each object track and from the extracted object track data for the object track, data defining a respective refined three-dimensional region in the point cloud that is a refined estimate of the three-dimensional region of the point cloud that includes points that are measurements of the corresponding object comprises:
determining, from the extracted object track data for the object track, whether the object track corresponds to a static object or a dynamic object.

7. The method of claim 6 , wherein generating, for each object track and from the extracted object track data for the object track, data defining a respective refined three-dimensional region in the point cloud that is a refined estimate of the three-dimensional region of the point cloud that includes points that are measurements of the corresponding object further comprises:
in response to determining that the object track corresponds to a static object:
generating an aggregate representation of the object track that includes extracted points from all of the one or more point clouds in the second coordinate system; and
processing the aggregate representation using a static track auto labeling neural network to generate the data defining the refined region.

8. The method of claim 7 , wherein processing the aggregate representation using a static track auto labeling neural network to generate the data defining the refined region comprises:
identifying one of the initial three-dimensional regions for the object track;
generating a network input by transforming each of the extracted points from the second coordinate system to a third coordinate system that is centered at a particular point in the identified initial three-dimensional region; and
providing the network input as input to the static track auto labeling neural network.

9. The method of claim 8 , wherein the object detector also outputs a confidence score for each three-dimensional region and wherein identifying one of the initial three-dimensional regions for the object track comprises selecting, from the initial three-dimensional regions for the object track, the initial three-dimensional region with a highest confidence score.

10. The method of claim 8 , wherein the static track auto labeling neural network outputs data identifying a three-dimensional region in the third coordinate system, and wherein generating the data defining the refined region comprises transforming the data identifying the three-dimensional region into the second coordinate system.

11. The method of claim 6 , wherein generating, for each object track and from the extracted object track data for the object track, data defining a refined region in the point cloud that is a refined estimate of a region of the point cloud that corresponds to the object track further comprises:
in response to determining that the object track corresponds to a dynamic object:
generating, for each of the one or more point clouds in which the corresponding object appears, a respective representation of the object track from the extracted point from the point cloud; and
processing the respective representations using a dynamic track auto labeling neural network to generate, for each of the one or more point clouds in which the corresponding object appears, data defining the respective refined region in the point cloud.

12. The method of claim 11 , wherein generating, for each of the one or more point clouds in which the corresponding object appears, a respective representation of the object track comprises:
transforming each of the extracted points from the point cloud from the second coordinate system to a respective fourth coordinate system that is centered at a particular point in the initial three-dimensional region in the point cloud for the object track.

13. The method of claim 12 , wherein, for each of the one or more point clouds in which the corresponding object appears, the dynamic track auto labeling neural network outputs data identifying a three-dimensional region in the respective fourth coordinate system, and wherein generating the data defining the refined region comprises transforming the data identifying the three-dimensional region from the respective fourth coordinate system to the second coordinate system.

14. The method of claim 11 , wherein each respective representation also includes data specifying the initial three-dimensional region in the point cloud for the object track.

15. A system comprising one or more computers an

Assignees

• Waymo Llc

Inventors

• Qi Ruizhongtai
• Zhou Yin
• Anguelov Dragomir
• Sun Pei

Classifications

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title

• G06T2207/10028

  Range image; Depth image; 3D point clouds · CPC title

• G06T7/20

  Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51) · CPC title

• G06V20/56

  exterior to a vehicle by using sensors mounted on the vehicle · CPC title

• G06V20/10

  Terrestrial scenes (scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59) · CPC title