Method and system for semantic label generation using sparse 3D data

Patent metadata

Field	Value
Publication number	US-10983217-B2
Application number	US-201816206465-A
Country	US
Kind code	B2
Filing date	Nov 30, 2018
Priority date	Nov 30, 2018
Publication date	Apr 20, 2021
Grant date	Apr 20, 2021

Abstract

Methods and apparatuses for generating a frame of semantically labeled 2D data are described. A frame of sparse 3D data is generated from a frame of sparse 3D data. Semantic labels are assigned to the frame of dense 3D data, based on a set of 3D bounding boxes determined for the frame of sparse 3D data. Semantic labels are assigned to a corresponding frame of 2D data based on a mapping between the frame of sparse 3D data and the frame of 2D data. The mapping is used to map a 3D data point in the frame of dense 3D data to a mapped 2D data point in the frame of 2D data. The semantic label assigned to the 3D data point is assigned to the mapped 2D data point. The frame of semantically labeled 2D data, including the assigned semantic labels, is outputted.

First claim

The invention claimed is:

1. A method for generating a frame of semantically labeled two dimensional (2D) data, the method comprising:
receiving a frame of sparse three-dimensional (3D) data;
generating a frame of dense 3D data from the frame of sparse 3D data by:
encoding the frame of sparse 3D data into sparse 2D xyz arrays, each sparse 2D xyz array representing a respective coordinate value of 3D data points in the sparse 3D data;
generating a sparse 2D depth array from the sparse 2D xyz arrays;
performing depth completion to generate a dense 2D depth array from the sparse 2D depth array;
generating dense 2D xyz arrays from the sparse 2D xyz arrays, using information from the depth completion; and
generating the frame of dense 3D data by performing an inverse mapping on the dense 2D xyz arrays;

assigning semantic labels to the frame of dense 3D data, the semantic labels being assigned based on a set of 3D bounding boxes determined for the frame of sparse 3D data, wherein each data point in the frame of dense 3D data falling within a given 3D bounding box is assigned the semantic label associated with the given 3D bounding box;
assigning the semantic labels to a frame of 2D data that corresponds to the frame of sparse 3D data based on a mapping between the frame of sparse 3D data and the frame of 2D data, wherein the mapping is used to map a 3D data point in the frame of dense 3D data to a mapped 2D data point in the frame of 2D data, and wherein the semantic label assigned to the 3D data point is assigned to the mapped 2D data point; and
outputting the frame of semantically labeled 2D data, including the assigned semantic labels.

2. The method of claim 1 , wherein the method is repeated for a plurality of frames of sparse 3D data, to output a set of semantically labeled 2D data that comprises a corresponding plurality of frames of semantically labeled 2D data.

3. The method of claim 1 , wherein the frame of sparse 3D data is received from one of a LIDAR sensor and a camera.

4. The method of claim 1 , wherein the frame of 2D data is received by performing a projection of the frame of dense 3D data.

5. The method of claim 1 , further comprising, after assigning the semantic labels to the frame of 2D data:
filtering the frame of semantically labeled 2D data by applying a set of 2D bounding boxes, wherein any assigned label that does not agree with the 2D bounding boxes is discarded or relabeled.

6. A method for generating a frame of semantically labeled two dimensional (2D) data, the method comprising:
receiving a frame of sparse three-dimensional (3D) data;
generating a frame of dense 3D data from the frame of sparse 3D data by:
encoding the frame of sparse 3D data into a sparse 2D z array representing a z coordinate value of 3D data points in the sparse 3D data, wherein the sparse 2D z array is considered to be a sparse 2D depth array;
performing depth completion to generate a dense 2D depth array from the sparse 2D depth array; and
generating the frame of dense 3D data by performing back projection on the dense 2D depth array;

assigning semantic labels to the frame of dense 3D data, the semantic labels being assigned based on a set of 3D bounding boxes determined for the frame of sparse 3D data, wherein each data point in the frame of dense 3D data falling within a given 3D bounding box is assigned the semantic label associated with the given 3D bounding box;
assigning the semantic labels to a frame of 2D data that corresponds to the frame of sparse 3D data based on a mapping between the frame of sparse 3D data and the frame of 2D data, wherein the mapping is used to map a 3D data point in the frame of dense 3D data to a mapped 2D data point in the frame of 2D data, and wherein the semantic label assigned to the 3D data point is assigned to the mapped 2D data point; and
outputting the frame of semantically labeled 2D data, including the assigned semantic labels.

7. The method of claim 1 , wherein the set of 3D bounding boxes is generated for the frame of sparse 3D data using a 3D neural network.

8. The method of claim 6 , wherein the method is repeated for a plurality of frames of sparse 3D data, to output a set of semantically labeled 2D data that comprises a corresponding plurality of frames of semantically labeled 2D data.

9. The method of claim 6 , wherein the frame of sparse 3D data is received from one of a LIDAR sensor and a camera.

10. The method of claim 6 , wherein the frame of 2D data is received by performing a projection of the frame of dense 3D data.

11. The method of claim 6 , further comprising, after assigning the semantic labels to the frame of 2D data:
filtering the frame of semantically labeled 2D data by applying a set of 2D bounding boxes, wherein any assigned label that does not agree with the 2D bounding boxes is discarded or relabeled.

12. An apparatus for generating a frame of semantically labeled two dimensional (2D) data, the apparatus comprising:
a processing unit; and
a memory coupled to the processing unit, the memory containing machine-executable instructions that, when executed by the processing unit, causes the apparatus to:
receive a frame of sparse three-dimensional (3D) data;
generate a frame of dense 3D data from the frame of sparse 3D data by:
encoding the frame of sparse 3D data into sparse 2D xyz arrays, each sparse 2D xyz array representing a respective coordinate value of 3D data points in the sparse 3D data;
generating a sparse 2D depth array from the sparse 2D xyz arrays;
performing depth completion to generate a dense 2D depth array from the sparse 2D depth array;
generating dense 2D xyz arrays from the sparse 2D xyz arrays, using information from the depth completion; and
generating the frame of dense 3D data by performing an inverse mapping on the dense 2D xyz arrays;

assign semantic labels to the frame of dense 3D data, the semantic labels being assigned based on a set of 3D bounding boxes determined for the frame of sparse 3D data, wherein each data point in the frame of dense 3D data falling within a given 3D bounding box is assigned the semantic label associated with the given 3D bounding box;
assign the semantic labels to a frame of 2D data that corresponds to the frame of sparse 3D data based on a mapping between the frame of sparse 3D data and the frame of 2D data, wherein the mapping is used to map a 3D data point in the frame of dense 3D data to a mapped 2D data point in the frame of 2D data, and wherein the semantic label assigned to the 3D data point is assigned to the mapped 2D data point; and
output the frame of semantically labeled 2D data, including the assigned semantic labels.

13. The apparatus of claim 12 , wherein the instructions further cause the apparatus to output a set of semantically labeled 2D data that comprises a plurality of frames of semantically labeled 2D data.

14. The apparatus of claim 12 , wherein the frame of sparse 3D data is received from one of a LIDAR sensor and a camera.

15. The apparatus of claim 12 , wherein the frame of 2D data is received by performing a projection of the dense 3D data.

16. The apparatus of claim 12 , wherein the instructions further cause the apparatus to, after assigning the semantic labels to the frame of 2D data:
filter the frame of semantically labeled 2D data by applying a set of 2D bounding boxes, wherein any assigned label that does not agree with the 2D bounding boxes is discarded or relabeled.

17. The apparatus of claim 12 , wherein the set of 3D bounding boxes is generated for the frame of sparse 3D data using

Assignees

• Nezhadarya Ehsan
• Nabatchian Amirhosein
• Liu Bingbing
• Huawei Tech Co Ltd

Inventors

• Nezhadarya Ehsan
• Nabatchian Amirhosein
• Liu Bingbing

Classifications

• G06N3/045

  Combinations of networks · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/0895

  Weakly supervised learning, e.g. semi-supervised or self-supervised learning · CPC title

• G06T19/00 Primary

  Manipulating three-dimensional [3D] models or images for computer graphics · CPC title