Method and apparatus for recovering point cloud data

Patent metadata

Field	Value
Publication number	US-10970864-B2
Application number	US-201816232797-A
Country	US
Kind code	B2
Filing date	Dec 26, 2018
Priority date	Dec 29, 2017
Publication date	Apr 6, 2021
Grant date	Apr 6, 2021

Abstract

A method for generating a point cloud data recovery model includes: acquiring at least one 2D image associated with a first point cloud data frame; partitioning the first point cloud data frame into at least one point cloud data set based on attributes of objects in the 2D image; and for each point cloud data set: determining a matching image of the first point cloud data frame from the at least one 2D image; determining 3D position data of a pixel point in the matching image based on the first point cloud data frame and at least one second point cloud data frame; and using 2D position data and the 3D position data of corresponding pixel points in the matching image as training input data and output data of a training model to generate a point cloud data recovery model for the object corresponding to the point cloud data set.

First claim

What is claimed is:

1. A method for generating a point cloud data recovery model, comprising:
acquiring at least one two-dimensional (2D) image associated with a first point cloud data frame;
partitioning the first point cloud data frame into at least one point cloud data set based on attributes of objects in the 2D image; and
for each point cloud data set,
determining, in the at least one 2D image, a matching image of the first point cloud data frame, the matching image matching the first point cloud data frame at at least one of a data acquisition position or data acquisition time;
determining three-dimensional (3D) position data of a pixel point in the matching image based on the first point cloud data frame and at least one second point cloud data frame associated with the first point cloud data frame; and
using 2D position data and the 3D position data of corresponding pixel points in the matching image as training input data and training output data of a training model, and generating a point cloud data recovery model for an object corresponding to the point cloud data set through a deep learning network.

2. The method according to claim 1 , wherein the partitioning comprises:
performing semantic segmentation on the at least one 2D image, to determine objects corresponding to pixel points in the at least one 2D image;
mapping point cloud data in the first point cloud data frame into a coordinate space of the at least one 2D image; and
aggregating the point cloud data in the first point cloud data frame based on the objects corresponding to the pixel points and the mapping, to form the at least one point cloud data set.

3. The method according to claim 1 , wherein the at least one 2D image is associated with the first point cloud data frame at at least one of the data acquisition position or the data acquisition time.

4. The method according to claim 1 , wherein the determining three-dimensional (3D) position data of a pixel point in the matching image comprises:
acquiring one or more point cloud data frames adjacent to the first point cloud data frame to be used as the at least one second point cloud data frame;
mapping the point cloud data in the first point cloud data frame and point cloud data in the second point cloud data frame into a coordinate space of the matching image; and
determining the 3D position data of the pixel point in the matching image based on the mapping.

5. A method for recovering point cloud data by using the point cloud data recovery model generated according to claim 1 , comprising:
acquiring at least one two-dimensional (2D) image associated with a to-be-recovered point cloud data frame;
partitioning the to-be-recovered point cloud data frame into at least one point cloud data set based on attributes of objects in the 2D image; and
for each point cloud data set,
determining, in the at least one 2D image, a matching image of the to-be-recovered point cloud data frame, the matching image matching the to-be-recovered point cloud data frame at at least one of a data acquisition position or data acquisition time; and
recovering respectively point cloud data in the point cloud data set respectively through the point cloud data recovery model for an object corresponding to the point cloud data set, based on 2D position data of a corresponding pixel point in the matching image.

6. The method according to claim 5 , wherein the partitioning comprises:
performing semantic segmentation on the at least one 2D image, to determine objects corresponding to pixel points in the at least one 2D image;
mapping point cloud data in the to-be-recovered point cloud data frame into a coordinate space of the at least one 2D image; and
aggregating the point cloud data in the to-be-recovered point cloud data frame based on the objects corresponding to the pixel points and the mapping, to form the at least one point cloud data set.

7. The method according to claim 5 , wherein the at least one 2D image is associated with the to-be-recovered point cloud data frame at at least one of the data acquisition position or the data acquisition time.

8. An apparatus for recovering point cloud data, comprising:
at least one processor; and
a memory storing instructions, the instructions when executed by the at least one processor, cause the at least one processor to perform operations according to claim 5 .

9. The apparatus according to claim 8 , wherein the partitioning comprises:
performing semantic segmentation on the at least one 2D image to determine objects corresponding to pixel points in the at least one 2D image;
mapping point cloud data in the to-be-recovered point cloud data frame into a coordinate space of the at least one 2D image; and
aggregating the point cloud data in the to-be-recovered point cloud data frame based on the objects corresponding to the pixel points and the mapping, to form the at least one point cloud data set.

10. The apparatus according to claim 8 , wherein the at least one 2D image is associated with the to-be-recovered point cloud data frame at at least one of the data acquisition position or the data acquisition time.

11. A computer non-transitory readable storage medium storing a computer program, wherein the program, when executed by a processor, cause the processor to implement the method according to claim 5 .

12. A computer non-transitory readable storage medium storing a computer program, wherein the program, when executed by a processor, cause the processor to implement the method according to claim 1 .

13. An apparatus for generating a point cloud data recovery model, comprising:
at least one processor; and
a memory storing instructions, the instructions when executed by the at least one processor, cause the at least one processor to perform operations, the operations comprising:
acquiring at least one two-dimensional (2D) image associated with a first point cloud data frame;
partitioning the first point cloud data frame into at least one point cloud data set based on attributes of objects in the 2D image;
determining, for each point cloud data set, a matching image of the first point cloud data frame from the at least one 2D image, the matching image matching the first point cloud data frame at at least one of a data acquisition position or data acquisition time;
determining, for the each point cloud data set, three-dimensional (3D) position data of a pixel point in the matching image based on the first point cloud data frame and at least one second point cloud data frame associated with the first point cloud data frame; and
using, for the each point cloud data set, 2D position data and the 3D position data of corresponding pixel points in the matching image as training input data and training output data of a training model, and generating a point cloud data recovery model for an object corresponding to the point cloud data set through a deep learning network.

14. The apparatus according to claim 13 , wherein the partitioning comprises:
performing semantic segmentation on the at least one 2D image, to determine objects corresponding to pixel points in the at least one 2D image;
mapping point cloud data in the first point cloud data frame into a coordinate space of the at least one 2D image; and
aggregating the point cloud data in the first point cloud data frame based on the objects corresponding to the pixel points and the mapping, to form the at least one point cloud data set.

15. The apparatus according to claim 13 , wherein the at least one 2D image is associated with the first point cloud data frame at at least one of the dat

Assignees

• Baidu online network technology beijing co ltd

Inventors

• Yan Miao
• Zhan Yifei
• Zhou Wang
• Duan Xiong
• Lang Xianpeng
• Ma Changjie

Classifications

• G06V10/764

  using classification, e.g. of video objects · CPC title

• G06V20/647 Primary

  by matching two-dimensional images to three-dimensional objects · CPC title

• G06T7/50 Primary

  Depth or shape recovery · CPC title

• G06F18/2163

  Partitioning the feature space · CPC title

• G06F18/2413

  based on distances to training or reference patterns · CPC title