Near real-time reconstruction using drones

Patent metadata

Field	Value
Publication number	US-12487607-B2
Application number	US-202117220046-A
Country	US
Kind code	B2
Filing date	Apr 1, 2021
Priority date	Apr 1, 2020
Publication date	Dec 2, 2025
Grant date	Dec 2, 2025

Abstract

Systems and methods for automatically constructing a 3-dimensional (3D) model of a feature using a drone. The method includes generating a reconnaissance flight path that minimizes battery usage by the drone, and conducting a discovery flight that uses the reconnaissance flight path. The method further includes transmitting reconnaissance laser sensor data from the drone to a processing system for target identification, and selecting a target feature for 3D model construction based on the reconnaissance laser sensor data. The method further includes scanning the target feature using a laser sensor, transmitting laser sensor data for the target feature having a minimum point density from the drone to the processing system for 3D model construction, and constructing the 3D model from the minimum point density laser sensor data.

First claim

What is claimed is:

1 . A method for automatically constructing a 3-dimensional 3D) model of a feature using a drone, comprising:
generating a reconnaissance flight path that minimizes battery usage by the drone that includes estimating drift of the drone for the reconnaissance flight path and calculating a correction to recalibrate the reconnaissance flight path of the drone, while scanning a target feature by:
estimating a pose of the drone in each frame from compressed light detection and ranging (LIDAR) frames obtained from the drone by continuously executing 3D simultaneous localization and mapping (SLAM) to obtain a SLAM trajectory;
aligning the SLAM trajectory with a global positioning system (GPS) trajectory by determining rigid transformation matrices of the SLAM trajectory and the GPS trajectory having synchronized timestamps;
minimizing drift of the drone by resetting the reconnaissance flight path of the drone to return to its starting position when a computed root mean square error RMSE of the SLAM trajectory and the GPS trajectory is greater than a drift threshold;
conducting a discovery flight that uses the reconnaissance flight path; transmitting reconnaissance laser sensor data from the drone to a processing system for target identification; selecting a target feature for 3D model construction based on the reconnaissance laser sensor data; scanning the target feature using a laser sensor; transmitting laser sensor data for the target feature having a minimum point density from the drone to the processing system for 3D model construction; and constructing the 3D model from the minimum point density laser sensor data.

2 . The method as recited in claim 1 , wherein the target feature is selected based on minimizing an objective function.

3 . The method as recited in claim 1 , wherein the laser sensor is LIDAR.

4 . The method as recited in claim 3 , further comprising compressing point cloud data collected by the LIDAR before transmitting the compressed point cloud data to the processing system.

5 . The method as recited in claim 1 , wherein scanning the target feature is conducted with the LIDAR aligned parallel with a direction of travel for a longitudinal flight path of the drone.

6 . The method as recited in claim 1 , wherein scanning the target feature starts from an origin and returns to the origin at an end of the reconnaissance flight path.

7 . A system for automatically constructing a 3-dimensional (3D) model of a feature using a drone, comprising:
the drone, including onboard electronics and light detection and ranging (LIDAR); and a computing system including: a trajectory planner configured to generate a flight trajectory for the drone that scans an entire area of interest and minimizes flight duration; a feature detector configured to identify individual visual features within a predefined area of interest; a 3D Simultaneous Localization And Mapping (SLAM) configured to generate a 3D model of one or more identified visual features from received point cloud data, and a drift estimator and recalibrator configured to estimate drift of the drone for a reconnaissance flight path and calculate corrections to recalibrate the reconnaissance flight path of the drone by:
estimating a pose of the drone in each frame from compressed LIDAR frames obtained from the drone by continuously executing 3D SLAM to obtain a SLAM trajectory;
aligning the SLAM trajectory with a global positioning system (GPS) trajectory by determining rigid transformation matrices of the SLAM trajectory and the GPS trajectory having synchronized timestamps;
minimizing drift of the drone by resetting the reconnaissance flight path of the drone to return to its starting position when a computed root mean square error (RMSE) of the SLAM trajectory and the GPS trajectory is greater than a drift threshold.

8 . The system of claim 7 , wherein the trajectory planner is further configured to generate the reconnaissance flight path that minimizes battery usage by the drone.

9 . The system of claim 7 , wherein the drift estimator and recalibrator is further configured to detect accumulated drift in position by the drone over the reconnaissance flight path, and correct for the accumulated drift in real-time.

10 . The system of claim 7 , wherein the feature detector is configured to identify individual visual features by detecting planar surfaces and estimating building height and boundaries from a plane surface forming a rooftop.

11 . The system of claim 7 , wherein the computing system receives compressed point clouds over a long-term evolution (LTE) network.

12 . The system of claim 11 , wherein the compressed point clouds utilize a communication bandwidth less than a bandwidth of the LTE network.

13 . A non-transitory computer readable storage medium comprising a computer readable program automatically constructing a 3-dimensional (3D) model of a feature using a drone, wherein the computer readable program when executed on a computer causes the computer to:
generate a reconnaissance flight path that minimizes battery usage by a drone that includes estimating drift of the drone for the reconnaissance flight path, and calculating a correction to recalibrate the reconnaissance flight path of the drone, while scanning a target feature by:
estimating a pose of the drone in each frame from compressed light detection and ranging (LIDAR) frames obtained from the drone by continuously executing 3D simultaneous localization and mapping SLAM) to obtain a SLAM trajectory;
aligning the SLAM trajectory with a global positioning system (GPS) trajectory by determining rigid transformation matrices of the SLAM trajectory and the GPS trajectory having synchronized timestamps;
minimizing drift of the drone by resetting the reconnaissance flight path of the drone to return to its starting position when a computed root mean square error RMSE of the SLAM trajectory and the GPS trajectory is greater than a drift threshold;
communicate a reconnaissance flight path for a discovery flight to the drone; receive reconnaissance laser sensor data from the drone for target identification; select a target feature for 3D model construction based on the reconnaissance laser sensor data; transmit coordinates for a scanning flight path for the target feature to the drone; receive laser sensor data for the target feature having a minimum point density from the drone for 3D model construction; and construct the 3D model from the minimum point density laser sensor data.

14 . The non-transitory computer readable storage medium comprising a computer readable program, as recited in claim 13 , wherein the target feature is selected based on minimizing an objective function.

15 . The non-transitory computer readable storage medium comprising a computer readable program, as recited in claim 13 , wherein the reconnaissance laser sensor data include compressed point clouds received over a long-term evolution (LTE) network.

16 . The non-transitory computer readable storage medium comprising a computer readable program, as recited in claim 15 , wherein the compressed point clouds utilize a communication bandwidth less than a bandwidth of the LTE network.

17 . The non-transitory computer readable storage medium comprising a computer readable program, as recited in claim 16 , wherein the scanning flight path is a longitudinal flight path that starts from an origin and returns to the origin at an end of the reconnaissance flight path.

18 . The method of claim 1

Assignees

• Nec Lab America Inc
• Nec Corp

Inventors

• Chai Eugene
• Sundaresan Karthikeyan
• Rangarajan Sampath

Classifications

• G01S7/4817

  relating to scanning · CPC title

• G05D1/46

  Control of position or course in three dimensions [3D] · CPC title

• G05D1/689

  Pointing payloads towards fixed or moving targets (positioning towed, pushed or suspended implements G05D1/672) · CPC title

• G01S17/894

  Three-dimensional [3D] imaging with simultaneous measurement of time-of-flight at a two-dimensional [2D] array of receiver pixels, e.g. time-of-flight cameras or flash lidar · CPC title

• B64U2201/20

  Remote controls · CPC title