System and method for upsampling of sparse point cloud for 3D registration

Patent metadata

Field	Value
Publication number	US-9972067-B2
Application number	US-201615290429-A
Country	US
Kind code	B2
Filing date	Oct 11, 2016
Priority date	Oct 11, 2016
Publication date	May 15, 2018
Grant date	May 15, 2018

Abstract

A method for three-dimensional point cloud registration includes generating a first upsampled three-dimensional point cloud by identifying at least one missing point in the three-dimensional point cloud, determining an intensity of neighboring pixels, filling the at least one missing point in the three-dimensional point cloud with a filler point using depth information from depth values in the three-dimensional point cloud that correspond with the neighboring pixels, generating a second upsampled three-dimensional point cloud by determining at least one local area of the first upsampled three-dimensional point cloud, determining entropies of pixels in the two-dimensional image that correspond with the at least one local area, adding at least one point to the at least one local area based on the entropies of pixels in the two-dimensional image and a scaled entropy threshold, and registering the second upsampled three-dimensional point cloud with a predetermined three-dimensional model.

First claim

What is claimed is:

1. A method for three-dimensional point cloud registration, the method comprising:
generating, with a processor, a three-dimensional point cloud of a scanned object using data received from a three-dimensional imaging sensor and storing the three-dimensional point cloud in a first memory location;
generating, with the processor, a two-dimensional image of the scanned object using data received from a two-dimensional imaging sensor and storing the two-dimensional image in a second memory location;
comparing the three-dimensional point cloud and the two-dimensional image, with the processor, and aligning the three-dimensional point cloud with the two-dimensional image according to at least three common points that are common to both the three-dimensional point cloud and the two-dimensional image;
generating, with the processor, a first upsampled three-dimensional point cloud by
identifying, with the processor, at least one missing point in the three-dimensional point cloud,
determining, with the processor, an intensity of neighboring pixels in the two-dimensional image neighboring the at least one missing point in the three-dimensional point cloud, and
filling, with the processor, the at least one missing point in the three-dimensional point cloud with a filler point using depth information from depth values in the three-dimensional point cloud that correspond with the neighboring pixels in the two-dimensional image where the neighboring pixels have intensities that correspond with an intensity of a point in the two-dimensional image associated with a respective missing point in the three-dimensional point cloud;

generating, with the processor, a second upsampled three-dimensional point cloud by
determining, with the processor, at least one local area of the first upsampled three-dimensional point cloud,
determining, with the processor, entropies of pixels in the two-dimensional image that correspond with the at least one local area of the first upsampled three-dimensional point cloud, and
adding, with the processor, at least one point to the at least one local area of the first upsampled three-dimensional point cloud based on the entropies of pixels in the two-dimensional image that correspond with the at least one local area and a scaled entropy threshold; and

registering, with the processor, the second upsampled three-dimensional point cloud with a predetermined three-dimensional model of the scanned object.

2. The method of claim 1 , wherein the at least one missing point is filled with the filler point when the intensities of the neighboring pixels and the intensity of the point in the two-dimensional image associated with the respective missing point in the three-dimensional point cloud match within a predetermined threshold.

3. The method of claim 1 , wherein filling the at least one missing point in the three-dimensional point cloud with the filler point further comprises assigning a weight for the neighboring pixels in the two-dimensional image.

4. The method of claim 3 , further comprising determining the weight for a respective neighboring pixel based on a distance between the at least one missing point and the respective neighboring pixel and a difference in intensity between the at least one missing point and the respective neighboring pixel.

5. The method of claim 1 , wherein generating the first upsampled three-dimensional point cloud further comprises:
assigning depth values to points in the two-dimensional image based on corresponding points in the three-dimensional point cloud; and
interpolating depth values for pixels of the two-dimensional image that do not have corresponding points in the three-dimensional point cloud.

6. The method of claim 5 , wherein interpolating the depth values for pixels of the two dimensional point cloud occurs in a predefined upsampling window around a point in the two-dimensional image having a known depth.

7. The method of claim 1 , wherein the at least one point added to the at least one local area of the first upsampled three-dimensional point cloud is located diagonally to pixels having a known depth.

8. The method of claim 1 , wherein the scaled entropy threshold includes:
a first threshold where upsampling does not occur if the entropies of pixels in the two-dimensional image that correspond with the at least one local area exceeds the first threshold;
a second threshold where a mean upsampling occurs if the entropies of pixels in the two-dimensional image that correspond with the at least one local area are less than the second threshold; and
a weighted upsampling occurs if the entropies of pixels in the two-dimensional image that correspond with the at least one local area is between the first and second thresholds.

9. An apparatus for three-dimensional point cloud registration, the apparatus comprising:
a three-dimensional imaging sensor;
a two-dimensional imaging sensor; and
a processor connected to both the three-dimensional sensor and the two-dimensional sensor, the processor being configured to:
generate a three-dimensional point cloud of a scanned object using data received from the three-dimensional imaging sensor and store the three-dimensional point cloud in a first memory location;
generate a two-dimensional image of the scanned object using data received from the two-dimensional imaging sensor and store the two-dimensional image in a second memory location;
compare the three-dimensional point cloud and the two-dimensional image and align the three-dimensional point cloud with the two-dimensional image according to at least one common point that is common to both the three-dimensional point cloud and the two-dimensional image;
generate a first upsampled three-dimensional point cloud by
identifying at least one missing point in the three-dimensional point cloud,
determining an intensity of neighboring pixels in the two-dimensional image neighboring the at least one missing point in the three-dimensional point cloud, and
filling the at least one missing point in the three-dimensional point cloud with a filler point using depth information from depth values in the three-dimensional point cloud that correspond with the neighboring pixels in the two-dimensional image where the neighboring pixels have intensities that correspond with an intensity of a point in the two-dimensional image associated with a respective missing point in the three-dimensional point cloud;

generate a second upsampled three-dimensional point cloud by
determining at least one local area of the first upsampled three-dimensional point cloud,
determining entropies of pixels in the two-dimensional image that correspond with the at least one local area of the first upsampled three-dimensional point cloud, and
adding at least one point to the at least one local area of the first upsampled three-dimensional point cloud based on the entropies of pixels in the two-dimensional image that correspond with the at least one local area and a scaled entropy threshold; and

registering the second upsampled three-dimensional point cloud with a predetermined three-dimensional model of the scanned object.

10. The apparatus of claim 9 , wherein the processor is configured to fill the at least one missing point with the filler point when the intensities of the neighboring pixels and the intensity of the point in the two-dimensional image associated with the respective missing point in the three-dimensional point cloud match within a predetermined threshold.

11. The apparatus of claim 9 , wherein filling the at least one missing point in the three-dimensional point cloud with the filler point further comprises assigning a weight for

Assignees

• Boeing Co

Inventors

• Kwon Hyukseong
• Kim Kyungnam

Classifications

• G06T7/0051

  Physics · mapped topic

• G06T3/0068 Primary

  Physics · mapped topic

• G06T3/4007

  based on interpolation, e.g. bilinear interpolation (image demosaicing G06T3/4015; edge-driven or edge-based scaling G06T3/403) · CPC title

• G06T2207/20148

  Physics · mapped topic

• G06T2207/10028

  Range image; Depth image; 3D point clouds · CPC title