Method and device for inpainting of colourised three-dimensional point clouds

Patent metadata

Field	Value
Publication number	US-11568520-B2
Application number	US-201815879303-A
Country	US
Kind code	B2
Filing date	Jan 24, 2018
Priority date	Jan 24, 2017
Publication date	Jan 31, 2023
Grant date	Jan 31, 2023

Abstract

A method for colourising a three-dimensional point cloud including surveying a point cloud with a surveying instrument. Each point of the point cloud may be characterised by coordinates within an instrument coordinate system having an instrument center. The method may include capturing a first image of the setting with a first camera. Each pixel value of the first image is assigned coordinates within a first camera coordinate system having a first projection center as origin and a first parallax shift relative to the instrument center. The method may include transforming the point cloud from the instrument coordinate system into the first camera coordinate system, resulting in a first transformed point cloud, detecting one or more uncovered points within the first transformed point cloud which are openly visible from the first projection center, and for each uncovered point, assigning a pixel value having corresponding coordinates in the first camera coordinate system.

First claim

What is claimed is:

1. A method for colourising a three-dimensional point cloud, the method comprising:
with a surveying instrument, surveying a point cloud of a setting by using a laser beam and associating a distance measured with the laser beam with an alignment of the laser beam under which the distance was measured, wherein each point of said point cloud is characterised by coordinates within an instrument coordinate system, which has an instrument center as an origin, the instrument center being defined by a crossing point of an azimuth axis and the elevation axis of the surveying instrument;
with a first camera of the surveying instrument, capturing a first image of the setting, wherein each pixel value of the first image is assigned to coordinates within a first camera coordinate system, which is provided with respect to a first projection center defining a perspective of the first camera onto the setting, wherein the first projection center has a first parallax shift relative to the instrument center;
transforming, with a computer, the point cloud from the instrument coordinate system into the first camera coordinate system, resulting in a first transformed point cloud,
within the first transformed point cloud, discriminating between uncovered points, which are openly visible from the perspective of the first projection center, and covered points, which are non-visible from the perspective of the first projection center due to the first parallax shift,
to each uncovered point in the first transformed point cloud, assigning a pixel value, which has corresponding coordinates in the first camera coordinate system, and
to each covered point in the first transformed point cloud, assigning a substitute pixel value, which is determined based on a pixel value assigned to a point adjacent to the covered point of the first transformed point cloud, wherein the point cloud is surveyed by means of the laser beam and the substitute pixel value is determined based on a property derived from a reflected part of the laser beam, wherein the substitute pixel value is determined by use of a signal-to-noise ratio provided by the reflected part of the laser beam.

2. The method according to claim 1 , further comprising:
with a second camera comprised by the surveying instrument, capturing a second image of the setting, wherein each pixel value of the second image is assigned to coordinates within a second camera coordinate system, which has a second projection center as origin, wherein the second projection center has second parallax shift relative to the instrument center,
transforming the point cloud from the instrument coordinate system into the second camera coordinate system, resulting in a second transformed point cloud,
within the second transformed point cloud, detecting one or more uncovered points, which are openly visible from the perspective of the second projection center,
to each uncovered point in the second transformed point cloud, assigning a pixel value, which has corresponding coordinates in the second camera coordinate system.

3. The method according to claim 2 , comprising:
within the second transformed point cloud, detecting one or more covered points, which are non-visible from the perspective of the second projection center due to the second parallax shift, and
to each covered point of the second transformed point cloud, assigning a substitute pixel value, which is determined based on a pixel value assigned to a point adjacent to the covered point of the second transformed point cloud.

4. The method according to claim 1 , further comprising:
with a second camera comprised by the surveying instrument, capturing a second image of the setting, wherein each pixel value of the second image is assigned to coordinates within a second camera coordinate system which has a second projection center as origin, wherein the second projection center has second parallax shift relative to the instrument center; and
transforming the point cloud from the instrument coordinate system into the second camera coordinate system, resulting in a second transformed point cloud,
within the first transformed point cloud, detecting one or more covered points, which are non-visible from the perspective of the first projection center due to the first parallax shift,
within the second transformed point cloud, determining corresponding points of said one or more covered points of the first transformed point cloud,
within the corresponding points, detecting one or more uncovered corresponding points, which are openly visible from the perspective of the second projection center, and
to each covered point in the first transformed point cloud, of which an uncovered corresponding point exists in the second transformed point cloud, assigning a pixel value of the second image that has corresponding coordinates with the uncovered corresponding point of the second transformed point cloud.

5. The method according to claim 1 , wherein at least one of the first camera and a second camera is one of a wide angle camera, a panoramic camera, and a spherical camera.

6. The method according to claim 2 , wherein at least one of the coordinates of the pixel values, the coordinates of the points of the point cloud, the coordinates of the points of the first transformed point cloud, and the coordinates of the points of the second transformed point cloud comprise at least an elevation angle and an azimuth angle.

7. The method according to claim 2 , wherein at least one of the coordinates of the points of the point cloud, the coordinates of the points of the first transformed point cloud, and the coordinates of the points of the second transformed point cloud comprise an elevation angle, an azimuth angle and a distance value.

8. The method according to claim 1 , wherein each pixel value of the first image is assigned to coordinates within the first camera coordinate system based at least on a focal length of the first camera.

9. The method according to claim 2 , wherein each pixel value of the second image is assigned to coordinates within the second camera coordinate system based at least on a focal length of the second camera.

10. The method according to claim 1 , wherein at least one of the uncovered points and the covered points are detected based on a detection algorithm using one of 3D point projection, plane detection, feature detection and object detection.

11. The method according to claim 1 , wherein the surveying instrument is configured for generating point clouds, wherein each point of the point clouds is characterised by coordinates within the instrument coordinate system having the instrument center as the origin, the surveying instrument comprising:
a base;
a body mounted on the base such that the body is rotatable relative to the base about an azimuth axis;
a beam directing unit mounted in the body such that the beam directing unit is rotatable relative to the body about an elevation axis, wherein the beam directing unit is configured to:
direct a transmission beam towards the setting, and
receive a reception beam from the setting;

the first camera having a first projection center, which has the first parallax shift relative to the instrument center, the first camera being configured to capture the first image of the setting, wherein each pixel value of the first image is assigned to coordinates within the first camera coordinate system, which has the first projection center as origin, wherein the first projection center has the first parallax shift relative to the instrument center; and
the computer is configured to the body, the beam directing unit and the first camera, the computer being configured to:

Assignees

• Leica Geosystems Ag

Inventors

• Blaser Daniel
• Fetzel Richard
• Gordon Bianca
• Schröder Marco
• Walser Bernd

Classifications

• G06T2207/10004

  Still image; Photographic image · CPC title

• G06T2207/10024

  Color image · CPC title

• G06T2207/10028

  Range image; Depth image; 3D point clouds · CPC title

• G01C15/002

  Active optical surveying means (optical plumbing G01C15/105) · CPC title

• G06T5/005 Primary

  Physics · mapped topic