Semantic labeling of point clouds using images

Patent metadata

Field	Value
Publication number	US-11580328-B1
Application number	US-202117188079-A
Country	US
Kind code	B1
Filing date	Mar 1, 2021
Priority date	Jul 21, 2017
Publication date	Feb 14, 2023
Grant date	Feb 14, 2023

Abstract

Systems and methods for semantic labeling of point clouds using images. Some implementations may include obtaining a point cloud that is based on lidar data reflecting one or more objects in a space; obtaining an image that includes a view of at least one of the one or more objects in the space; determining a projection of points from the point cloud onto the image; generating, using the projection, an augmented image that includes one or more channels of data from the point cloud and one or more channels of data from the image; inputting the augmented image to a two dimensional convolutional neural network to obtain a semantic labeled image wherein elements of the semantic labeled image include respective predictions; and mapping, by reversing the projection, predictions of the semantic labeled image to respective points of the point cloud to obtain a semantic labeled point cloud.

First claim

What is claimed is:

1. A system, comprising:
a data processing apparatus; and
a data storage device storing instructions executable by the data processing apparatus that upon execution by the data processing apparatus cause the data processing apparatus to perform operations comprising:
obtaining a semantic labeled point cloud, wherein points of the semantic labeled point cloud include respective predictions for the points;
determining a graph based on the semantic labeled point cloud, wherein nodes of the graph are points from the semantic labeled point cloud and edges of the graph connect nodes with respective points that satisfy a pairwise criteria;
identifying one or more connected components of the graph; and
determining clusters of points from the semantic labeled point cloud by performing a hierarchical segmentation of each of the one or more connected components of the graph.

2. The system of claim 1 , wherein the operations comprise:
inputting predictions based on predictions for points of one of the clusters to a three dimensional convolutional neural network to obtain a prediction for the cluster; and
assigning the prediction for the cluster to all points of the cluster in the semantic labeled point cloud.

3. The system of claim 2 , wherein the predictions input to the three dimensional convolutional neural network are associated with respective voxels that collectively form a block centered at a center of the one of the clusters, and wherein the predictions input to the three dimensional convolutional neural network are determined as an average of predictions for points located within a respective voxel.

4. The system of claim 1 , wherein the operations comprise:
applying a fully connected conditional random field to the predictions of the semantic labeled point cloud to refine the predictions.

5. The system of claim 1 , wherein the operations comprise:
determining a difference between respective values for two points from the semantic labeled point cloud; and
checking the difference meets a threshold to evaluate the pairwise criteria for the two points.

6. The system of claim 5 , wherein the difference is between respective positions for the two points.

7. The system of claim 1 , wherein the hierarchical segmentation is a Felzenszwalb segmentation.

8. A method comprising:
obtaining a semantic labeled point cloud, wherein points of the semantic labeled point cloud include respective predictions for the points;
determining a graph based on the semantic labeled point cloud, wherein nodes of the graph are points from the semantic labeled point cloud and edges of the graph connect nodes with respective points that satisfy a pairwise criteria;
identifying one or more connected components of the graph; and
determining clusters of points from the semantic labeled point cloud by performing a hierarchical segmentation of each of the one or more connected components of the graph.

9. The method of claim 8 , comprising:
inputting predictions based on predictions for points of one of the clusters to a three dimensional convolutional neural network to obtain a prediction for the cluster; and
assigning the prediction for the cluster to all points of the cluster in the semantic labeled point cloud.

10. The method of claim 9 , wherein the predictions input to the three dimensional convolutional neural network are associated with respective voxels that collectively form a block centered at a center of the one of the clusters, and wherein the predictions input to the three dimensional convolutional neural network are determined as an average of predictions for points located within a respective voxel.

11. The method of claim 8 , comprising:
applying a fully connected conditional random field to the predictions of the semantic labeled point cloud to refine the predictions.

12. The method of claim 8 , comprising:
determining a difference between respective values for two points from the semantic labeled point cloud; and
checking the difference meets a threshold to evaluate the pairwise criteria for the two points.

13. The method of claim 12 , wherein the difference is between respective lidar intensities for the two points.

14. The method of claim 12 , wherein the difference is between respective colors for the two points.

15. The method of claim 12 , wherein the difference is between respective positions for the two points.

16. The method of claim 8 , wherein the hierarchical segmentation is a Felzenszwalb segmentation.

17. A non-transitory computer-readable storage medium including program instructions executable by one or more processors that, when executed, cause the one or more processors to perform operations, the operations comprising:
determining a projection of points from a point cloud onto an image;
generating, using the projection, an augmented image that includes one or more channels of data from the point cloud and one or more channels of data from the image;
inputting the augmented image to a neural network to obtain a semantic labeled image, wherein elements of the semantic labeled image include respective predictions;
mapping, by reversing the projection, predictions of the semantic labeled image to respective points of the point cloud to obtain a semantic labeled point cloud; and
determining a graph based on the semantic labeled point cloud, wherein nodes of the graph are points from the semantic labeled point cloud and edges of the graph connect nodes with respective points that satisfy a pairwise criteria;
identifying one or more connected components of the graph; and
determining clusters of points from the semantic labeled point cloud by performing a hierarchical segmentation of each of the one or more connected components of the graph.

18. The non-transitory computer-readable storage medium of claim 17 , wherein the image is a first image and the semantic labeled image is a first semantic labeled image, and further including program instructions executable by one or more processors that, when executed, cause the one or more processors to perform operations, the operations comprising:
determining a second semantic labeled image based on a second image augmented with data from the point cloud;
mapping predictions of the second semantic labeled image to respective points of the point cloud; and
accumulating predictions from the first semantic labeled image and from the second semantic labeled image for at least one point of the semantic labeled point cloud.

19. The non-transitory computer-readable storage medium of claim 17 , further including program instructions executable by one or more processors that, when executed, cause the one or more processors to perform operations, the operations comprising:
searching a set of images associated with different respective camera locations to identify a subset of images that includes at least two images with views of each point in the point cloud; and
wherein the image is obtained from the subset of images.

20. The non-transitory computer-readable storage medium of claim 17 , wherein the point cloud is determined using a bundle adjustment process based on lidar scans captured at a plurality of locations and times, and further including program instructions executable by one or more processors that, when executed, cause the one or more processors to perform operations, the operations comprising:
assigning indications of moving likelihood to respective points of the point cloud based on how frequently the respective poin

Assignees

• Ho Huy Tho
• Wang Jingwei
• Larsson Kjell Fredrik
• Apple Inc

Inventors

• Ho Huy Tho
• Wang Jingwei
• Larsson Kjell Fredrik

Classifications

• G06T2207/10028

  Range image; Depth image; 3D point clouds · CPC title

• G06T7/11 Primary

  Region-based segmentation · CPC title

• G06V10/774

  Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting · CPC title

• G06T2200/28

  involving image processing hardware · CPC title

• G06T2207/20081

  Training; Learning · CPC title