Method for point cloud up-sampling based on deep learning

Patent metadata

Field	Value
Publication number	US-11880959-B2
Application number	US-202017418366-A
Country	US
Kind code	B2
Filing date	Oct 30, 2020
Priority date	May 19, 2020
Publication date	Jan 23, 2024
Grant date	Jan 23, 2024

Abstract

The present invention discloses a method for point cloud up-sampling based on deep learning, including: obtaining training data including a first number of sparse input points and a second number of dense input points; constructing a deep network model to be used for respectively performing replication and sampling operation based on curvature on initial eigenvectors extracted from the first number of sparse input points to obtain a second number of intermediate eigenvectors, performing splicing operation on each intermediate eigenvector, inputting the spliced intermediate eigenvectors into a multilayer perceptron, and determining sampling prediction points based on the sampling eigenvectors output by the multilayer perceptron; training the deep network model until an objective function determined by the sampling prediction points and the dense input points converges; and testing the deep network model to obtain point cloud data of an object under test after up-sampling.

First claim

What is claimed is:

1. A method for point cloud up-sampling based on deep learning, comprising:
step 1: obtaining training data, wherein the training data comprises a first number of sparse input points and a second number of dense input points, the first number of sparse input points are uniformly sampled from a Computer Aided Design (CAD) model, and the second number of dense input points are sampled from the CAD model based on curvature;
step 2: constructing a deep network model, wherein the deep network model is used for extracting initial eigenvectors from the first number of sparse input points, respectively performing a replication operation and a sampling operation based on curvature features on the initial eigenvectors to obtain a second number of intermediate eigenvectors, performing a splicing operation on each intermediate eigenvector of the second number of intermediate eigenvectors, inputting the spliced intermediate eigenvectors into a multilayer perceptron to obtain a second number of sampling eigenvectors, and determining a second number of sampling prediction points according to the second number of sampling eigenvectors;
step 3: training the deep network model, determining an objective function according to the second number of sampling prediction points and the second number of dense input points, and adjusting parameters of the deep network model based on the objective function until the objective function converges; and
step 4: testing the deep network model, sampling several seed points from a point cloud representation of an object under test, respectively constructing an input point cloud to be tested from each seed point of the several seed points and neighborhood points thereof, inputting the input point clouds to be tested into the trained deep network model to obtain a test point cloud corresponding to each of the input point clouds to be tested respectively, and aggregating all of the test point clouds to obtain point cloud data of the object under test after up-sampling.

2. The method of claim 1 , wherein the deep network model comprises a feature extraction module, a feature sampling module and a coordinate regression module, wherein
the feature extraction module is configured to extract eigenvectors from all of the first number of sparse input points received by the deep network model to obtain the initial eigenvectors of the first number of sparse input points;
the feature sampling module is configured to replicate the-initial eigenvectors to obtain a third number of first eigenvectors, determine a sampling probability according to a curvature of each sparse input point of the first number of sparse input points, perform sampling according to the sampling probability to obtain a fourth number of second eigenvectors, determine the second number of intermediate eigenvectors according to all of the third number of first eigenvectors and all of the fourth number of second eigenvectors, splice a 2D vector generated by a 2D mesh mechanism onto each intermediate eigenvector of the second number of intermediate eigenvectors, and input the spliced intermediate eigenvectors into the multilayer perceptron to obtain the second number of sampling eigenvectors, wherein the 2D vectors spliced onto each intermediate eigenvector of the second number of intermediate eigenvectors are different; and
the coordinate regression module is configured to determine the second number of sampling prediction points according to the second number of sampling eigenvectors.

3. The method of claim 1 , wherein the determining an objective function according to the second number of sampling prediction points and the second number of dense input points comprises:
determining a reconstruction loss according to a distance between the second number of sampling prediction points and the second number of dense input points;
determining a curvature-aware loss according to a relationship between a curvature of a random point on a sampling prediction point cloud and an area of a neighborhood of the random point;
determining a smoothness loss according to a geometric relationship of the random point on the sampling prediction point cloud relative to a neighborhood of the random point on a dense input point cloud; and
determining the objective function according to the reconstruction loss, the curvature-aware loss and the smoothness loss;
wherein the sampling prediction point cloud consists of the second number of sampling prediction points, and the dense input point cloud consists of the second number of dense input points.

4. The method of claim 3 , wherein the determining the objective function according to the reconstruction loss, the curvature-aware loss and the smoothness loss comprises:
obtaining the objective function from a weighted sum of the reconstruction loss, the curvature-aware loss and the smoothness loss.

Assignees

• Univ South China Tech

Inventors

• Jia Kui
• Lin Jiehong
• Chen Ke

Classifications

• G06T3/4046 Primary

  using neural networks · CPC title

• G06F30/23

  using finite element methods [FEM] or finite difference methods [FDM] · CPC title

• G06T17/20 Primary

  Finite element generation, e.g. wire-frame surface description, {tesselation} · CPC title

• G06T17/205

  Re-meshing · CPC title

• G06T2207/10028

  Range image; Depth image; 3D point clouds · CPC title