Performing 3D reconstruction via an unmanned aerial vehicle

Patent metadata

Field	Value
Publication number	US-11940795-B2
Application number	US-202318099571-A
Country	US
Kind code	B2
Filing date	Jan 20, 2023
Priority date	Feb 13, 2020
Publication date	Mar 26, 2024
Grant date	Mar 26, 2024

Abstract

In some examples, an unmanned aerial vehicle (UAV) may include one or more processors configured to capture, with one or more image sensors, and while the UAV is in flight, a plurality of images of a target. The one or more processors may compare a first image of the plurality of images with a second image of the plurality of images to determine a difference between a current frame of reference position for the UAV and an estimate of an actual frame of reference position for the UAV. In addition, the one or more processors may determine, based at least on the difference, and while the UAV is in flight, an update to a three-dimensional model of the target.

First claim

What is claimed:

1. An unmanned aerial vehicle (UAV) comprising:
one or more image sensors;
a propulsion mechanism; and
one or more processors configured by executable instructions to:
receive, from a computing device, an indication of a target and one or more scan parameters for scanning the target;
capture, with the one or more image sensors, while the UAV is in flight, a plurality of images of the target;
compare a first image of the plurality of images with a second image of the plurality of images to determine a difference between a current frame of reference position for the UAV and an estimate of an actual frame of reference position for the UAV; and
determine, based at least on the difference, and while the UAV is in flight, an update to a three-dimensional (3D) model of the target.

2. The UAV as recited in claim 1 , wherein:
the 3D model of the target includes a plurality of points representative of a surface of the target; and
the one or more processors configured by the executable instructions to determine, based at least on the difference, the update to the 3D model of the target comprises the one or more processors further configured to determine at least one of:
an updated location of at least one point in the 3D model, or
a location of a new point in the 3D model.

3. The UAV as recited in claim 1 , the one or more processors further configured to use distance information determined based on the plurality of images for determining respective locations in 3D space of a plurality of points of the 3D model, the plurality of points representative of a surface of the target.

4. The UAV as recited in claim 1 , the one or more processors further configured to:
based at least in part on the update to the 3D model, update, by the one or more processors, and while the UAV is in flight, a scan plan for scanning at least a portion of the target, the scan plan including a plurality of poses for the UAV to assume to capture images of the target; and
control, by the one or more processors, a propulsion mechanism of the UAV to cause the UAV to fly to at least one of an additional pose or an updated pose included in the updated scan plan.

5. The UAV as recited in claim 1 , wherein comparing the first image of the plurality of images with the second image of the plurality of images to determine the difference further comprises the one or more processors configured to:
compare the first image with the second image to determine one or more correspondences between the first image and the second image; and
based at least in part on the one or more correspondences, determine, by the one or more processors, the difference between the current frame of reference position for the UAV and the estimate of the actual frame of reference position for the UAV.

6. The UAV as recited in claim 1 , the one or more processors further configured to:
capture a plurality of higher-resolution images of the target;
determine correspondences between the plurality of higher-resolution images based at least on a plurality of the differences between the current frame of reference positions for the UAV and the estimated actual frame of reference positions for the UAV for the plurality of higher-resolution images; and
generate, based at least in part on the determined correspondences, a higher-resolution 3D model that is of a higher resolution than the 3D model.

7. A method comprising:
capturing, by one or more processors of an unmanned aerial vehicle (UAV), with one or more image sensors, and while the UAV is in flight, a plurality of images of a target;
comparing, by the one or more processors, a first image of the plurality of images with a second image of the plurality of images to determine a difference between a current frame of reference position for the UAV and an estimate of an actual frame of reference position for the UAV; and
determining, by the one or more processors, based at least on the difference, and while the UAV is in flight, an update to a three-dimensional (3D) model of the target.

8. The method as recited in claim 7 , wherein:
the 3D model of the target includes a plurality of points representative of a surface of the target; and
the determining, based at least on the difference, the update to the 3D model of the target comprises determining at least one of:
an updated location of at least one point in the 3D model, or
a location of a new point in the 3D model.

9. The method as recited in claim 7 , further comprising using distance information determined based on the plurality of images for determining respective locations in 3D space of a plurality of points of the 3D model, the plurality of points representative of a surface of the target.

10. The method as recited in claim 7 , further comprising:
based at least in part on the update to the 3D model, updating, by the one or more processors, and while the UAV is in flight, a scan plan for scanning at least a portion of the target, the scan plan including a plurality of poses for the UAV to assume to capture images of the target; and
controlling, by the one or more processors, a propulsion mechanism of the UAV to cause the UAV to fly to at least one of an additional pose or an updated pose included in the updated scan plan.

11. The method as recited in claim 7 , wherein the comparing the first image of the plurality of images with the second image of the plurality of images to determine the difference further comprises:
comparing the first image with the second image to determine one or more correspondences between the first image and the second image; and
based at least in part on the one or more correspondences, determining, by the one or more processors, the difference between the current frame of reference position for the UAV and the estimate of the actual frame of reference position for the UAV.

12. The method as recited in claim 7 , further comprising:
grouping together a plurality of pairs of images of the plurality of images, including the first image and the second image; and
updating relative locations of multiple points of the 3D model based on comparing respective locations of respective features in the plurality of pairs of images.

13. The method as recited in claim 7 , further comprising:
capturing a plurality of higher-resolution images of the target;
determining correspondences between the plurality of higher-resolution images based at least on a plurality of the differences between the current frame of reference positions for the UAV and the estimated actual frame of reference positions for the UAV for the plurality of higher-resolution images; and
generating, based at least in part on the determined correspondences, a higher-resolution 3D model that is of a higher resolution than the 3D model.

14. An aerial vehicle comprising
one or more image sensors;
a propulsion mechanism; and
one or more processors configured by executable instructions while the aerial vehicle is in flight to:
direct the one or more image sensors to capture multiple images of a target;
compare a first image of the multiple images with a second image of the multiple images to identify a difference between a current frame of reference position for the aerial vehicle and an estimate of an actual frame of reference position for the aerial vehicle; and
determine, based at least on the difference, an update to a three-dimensional (3D) model of the target.

15. The aerial vehicle as recited in claim 14 , wherein:
the 3D model of the target includes a plurality of points representative of a surface of the target; and
for determining,

Assignees

• Skydio Inc

Inventors

• Henry Peter
• Zhu Jack
• Richman Brian
• Zheng Harrison
• Martirosyan Hayk
• Donahoe Matthew
• Bachrach Abraham
• Bry Adam
• Kennedy Ryan David
• Mondal Himel
• Delepine Quentin Allen Wah Yen

Classifications

• B64D31/06

  actuated automatically · CPC title

• B64D47/08

  Arrangements of cameras · CPC title

• B64C39/024

  of the remote controlled vehicle type, i.e. RPV · CPC title

• B64U10/13

  Flying platforms · CPC title

• B64U2201/20

  Remote controls · CPC title