Plant point cloud acquisition, registration and optimization method based on TOF camera

Patent metadata

Field	Value
Publication number	US-11113835-B2
Application number	US-202016885682-A
Country	US
Kind code	B2
Filing date	May 28, 2020
Priority date	Jun 6, 2019
Publication date	Sep 7, 2021
Grant date	Sep 7, 2021

Abstract

The present invention discloses a plant point cloud acquisition, registration, and optimization method based on a time of flight (TOF) camera, which includes the following steps: (1) placing a to-be-tested plant on a turntable, adjusting a view angle of the TOF camera, and aligning the TOF camera with the to-be-tested plant; (2) turning on the turntable so that it rotates automatically, and enabling the TOF camera to acquire point cloud data of the to-be-tested plant at intervals; (3) performing real-time preprocessing on each frame of point cloud data acquired by the TOF camera; (4) performing registration and optimization on every two adjacent frames of point cloud data, and then integrating the data to obtain complete plant point cloud data; and (5) using statistical filtering to remove the discrete noise in the plant point cloud data obtained in the registration and optimization process to obtain final point cloud data.

First claim

What is claimed is:

1. A plant point cloud acquisition, registration, and optimization method based on a time of flight (TOF) camera, comprising:
(1) placing a to-be-tested plant on a turntable, adjusting a view angle of the TOF camera, and aligning the TOF camera with the to-be-tested plant;
(2) turning on the turntable so that it rotates automatically, and enabling the TOF camera to acquire point cloud data of the to-be-tested plant at intervals;
(3) performing real-time preprocessing on each frame of point cloud data acquired by the TOF camera;
(4) performing registration and optimization on every two adjacent frames of point cloud data, and then integrating the data to obtain complete plant point cloud data, comprising:
(4-1) transforming a coordinate system of a second point cloud P 2 to a coordinate system of a first point cloud P 1 to obtain a point cloud P 2 ′;
(4-2) triangulating P 1 and P 2 ′, removing boundary points that do not form a triangular patch, but retaining points inside that do not form a triangular patch;
(4-3) searching for a triangular patch of P 2 ′ within the neighborhood of each triangular patch in P 1 , and checking whether the triangular patch in P 1 intersects with or is in parallel to the triangular patches within the neighborhood;
(4-4) according to the intersection and parallelism relationship, adjust a point cloud set of P 2 ′ that is within the neighborhood of the triangular patch in P 1 to obtain a point cloud P 2 ″;
(4-5) adding points in P 2 ′ that do not form a triangular patch to P 2 ″, and then performing downsampling to obtain a point set P 2 ′deal;
(4-6) transforming a coordinate system of an i-th point cloud Pi to the coordinate system of the first point cloud P 1 to obtain a point cloud Pi′, and repeating steps (4-2) to (4-5) to obtain Pi′deal, wherein i≥3; and
(4-7) integrating P 1 , P 2 ′deal, and Pi′deal to obtain the complete plant point cloud data; and
(5) using statistical filtering to remove the discrete noise in the plant point cloud data obtained in the registration and optimization process to obtain final point cloud data.

2. The plant point cloud acquisition, registration, and optimization method based on a TOF camera according to claim 1 , wherein step (1) comprises:
(1-1) calibrating the TOF camera with Zhang Zhengyou method, placing the to-be-tested plant on the turntable, adjusting the view angle of the TOF camera, and aligning the TOF camera with the to-be-tested plant; and
(1-2) using a plane fitting method to obtain a fitted plane of a tabletop, and obtaining an angle between a normal vector of the fitted plane of the tabletop and each axis of a right-hand coordinate system.

3. The plant point cloud acquisition, registration, and optimization method based on a TOF camera according to claim 2 , wherein step (1-2) comprises:
(1-2a) performing extreme pass-through filtering on each frame of point cloud data so that only part of the visible tabletop is retained;
(1-2b) obtaining multiple fitted planes through random sample consensus (RANSAC) plane fitting, obtaining a normal vector of each fitted plane, and determining the fitted plane of the tabletop based on an angle relationship between the normal vector of the fitted plane and the y-axis of the right-hand coordinate system and a point cloud quantity threshold, wherein the fitted plane of the tabletop meets the following formulas:
min{θ,θ=arcos( {right arrow over (n)} fitted plane , y )}  (1)

max{ n y ,n y ∈{n y i ,i= 1,2 . . . m}}   (2)

num plane ≥num threshold   (3)

wherein {right arrow over (n)} fitted plane is the normal vector of the fitted plane; y is the y-axis of the right-hand coordinate system; θ is the angle between the normal vector of the fitted plane and the y-axis of the right-hand coordinate system; n y i is a y-axis component of the normal vector of each fitted plane, m is a quantity of fitted planes, and n y is a y-axis component of a normal of the fitted plane; num plane is a quantity of point clouds of the fitted plane; and num threshold is the point cloud quantity threshold; and
(1-2c) after determining the fitted plane of the tabletop, obtaining the angle between the normal vector of the fitted plane of the tabletop and each axis of the right-hand coordinate system.

4. The plant point cloud acquisition, registration, and optimization method based on a TOF camera according to claim 1 , wherein step (3) comprises:
(3-1) rotating each frame of the point cloud data acquired by the TOF camera according to an angle between a normal vector of a fitted plane of a tabletop and each axis of a right-hand coordinate system;
(3-2) performing pass-through filtering on the rotated point cloud data to remove background and obtain an unordered point cloud; and
(3-3) using bilateral filtering to smoothen the unordered point cloud.

5. The plant point cloud acquisition, registration, and optimization method based on a TOF camera according to claim 1 , wherein the step (4-2) of searching for boundary points based on a KD tree and normal vector method comprises:
(4-2a) searching for neighboring points of a point that does not form a triangular patch through the KD tree, wherein when a quantity of neighboring points of a point is less than a threshold, the point is an outlier; and
(4-2b) according to the quantity of neighboring points of the point that does not form a triangular patch, using a PCA method to calculate a normal vector of the point, determining a connection line between a view point and the point, and calculating an angle between the connection line and the normal vector, wherein if the angle is greater than a threshold, the point is a flying pixel point; and both outliers and flying pixel points are boundary points.

6. The plant point cloud acquisition, registration, and optimization method based on a TOF camera according to claim 5 , wherein in step (4-4), if a triangular patch Δabc in P 1 is parallel to a triangular patch Δmnq of P 2 ′ within the neighborhood, then:
(I-1) calculating a distance between the Δabc plane and the Δmnq plane;
if the distance is greater than a preset moving distance threshold, skipping adjusting Δmnq;
if the distance is less than or equal to the preset moving distance threshold, calculating a distance between normals of Δabc and Δmnq that cross the centroids of the triangles; if the distance is greater than a preset threshold, skipping adjusting Δmnq; and if the distance is less than or equal to the preset threshold, calculating a median plane of Δabc and Δmnq, and projecting three vertices of Δmnq onto the median plane along a normal direction of the median plane, to obtain Δm′n′q; and
(I-2) iteratively calculating a distance between the Δabc plane and the Δm′n′q′ plane, and repeating step (I-1) until the distance is less than a final distance threshold to obtain a point cloud of a new triangular patch.

7. The plant point cloud acquisition, registration, and optimization method based on a TOF camera according to claim 5 , wherein in step (4-4), if a triangular patch Δabc in P 1 intersects with a triangular patch Δmnq of P 2 ′ within the neighborhood, then:
(II-1) calculating an angle between the Δabc plane and the Δmnq plane;
if the angle is greater than an initial angle threshold, skipping adjusting Δmnq;
if the angle is less than or equal to the initial angle threshold, calculating a median plane of Δabc and Δmnq, and projecting three vertices of Δmnq onto the median plane along a normal direction of the median plane to obtain Δm′n′q; and
(II-2) iteratively calculating the angle between the Δabc plane and the Δm′n′q′ plane, and repeating step (II-1) until the angle is less than a final angle threshold, to obtain a point cloud of

Assignees

• Univ Zhejiang

Inventors

• Cen Haiyan
• He Yong
• Ma Zhihong
• Fang Hui

Classifications

• G06V2201/12

  Acquisition of 3D measurements of objects · CPC title

• G06V20/64

  Three-dimensional [3D] objects · CPC title

• G06V20/188

  Vegetation · CPC title

• G06T7/60 Primary

  Analysis of geometric attributes · CPC title

• G06T7/30 Primary

  Determination of transform parameters for the alignment of images, i.e. image registration · CPC title