Aerial vehicle smart landing

Patent metadata

Field	Value
Publication number	US-11242144-B2
Application number	US-201916272111-A
Country	US
Kind code	B2
Filing date	Feb 11, 2019
Priority date	Feb 9, 2018
Publication date	Feb 8, 2022
Grant date	Feb 8, 2022

Abstract

A technique is introduced for autonomous landing by an aerial vehicle. In some embodiments, the introduced technique includes processing a sensor data such as images captured by onboard cameras to generate a ground map comprising multiple cells. A suitable footprint, comprising a subset of the multiple cells in the ground map that satisfy one or more landing criteria, is selected and control commands are generated to cause the aerial vehicle to autonomously land on an area corresponding to the footprint. In some embodiments, the introduced technique involves a geometric smart landing process to select a relatively flat area on the ground for landing. In some embodiments, the introduced technique involves a semantic smart landing process where semantic information regarding detected objects is incorporated into the ground map.

First claim

What is claimed is:

1. A method for autonomous landing by an unmanned aerial vehicle (UAV), the method comprising:
receiving, by a computer system, sensor data from a sensor onboard the UAV while the UAV is in flight through a physical environment, the sensor comprising an image capture device mounted to the UAV and the sensor data comprising images captured by the image capture device;
processing, by the computer system, the sensor data including the images to determine a plurality of data points representative of height values at a plurality of points along a surface in the physical environment;
generating and continually updating, by the computer system, a ground map of a portion of the surface based on the determined height values, wherein the ground map comprises a grid of a plurality of cells, wherein each cell includes continually updated height statistics based on the determined data points associated with the cell;
identifying, by the computer system, a footprint based on the ground map, the footprint including a subset of the plurality of cells that include height statistics that satisfy a specified criterion;
designating, by the computer system, a landing area on the surface in the physical environment based on the identified footprint; and
generating, by the computer system, control commands configured to cause the UAV to autonomously maneuver to land on the designated landing area.

2. The method of claim 1 , wherein generating the control commands includes:
generating a behavioral objective to land on the designated landing area; and
inputting the generated behavioral objective into a motion planner configured to process a plurality of behavioral objectives to generate a planned trajectory;
wherein the control commands are generated based on the planned trajectory generated by the motion planner.

3. The method of claim 1 , wherein processing the sensor data to determine the plurality of data points includes:
processing images captured by the image capture device to generate a disparity image; and
mapping pixels in the generated disparity image to three-dimensional (3D) points in space corresponding to the points along the surface in the physical environment, each of the 3D points having a height value based on its respective position in space;
wherein the data points are based on the height values of the 3D points.

4. The method of claim 3 , wherein generating the ground map includes:
continually adding data points to one or more cells in the ground map as new images are processed; and
updating the height statistics for the one or more cells as data points are added.

5. The method of claim 4 , wherein updating the height statistics for the one or more cells includes:
down-weighting height values associated with older data points as newer data points are added to a particular cell of the one or more cells.

6. The method of claim 1 , wherein the height statistics based on the determined data points for a particular cell include any of an average height, median height, minimum height, or maximum height of points along a portion of the surface corresponding to the particular cell.

7. The method of claim 1 , wherein the specified criterion is satisfied if the height statistics associated with the subset of the plurality of cells have a variance below a threshold level.

8. The method of claim 1 , further comprising:
processing the sensor data to detect a physical object in the physical environment and extract semantic information associated with the detected physical object; and
adding the semantic information to one or more cells in the ground map that correspond with a location of the physical object in the physical environment;
wherein the subset of the plurality of cells included in the identified footprint include semantic information that satisfies the specified criterion.

9. The method of claim 1 , further comprising:
before landing on the designated landing area:
continuing to process new sensor data received from the sensors to determine if autonomous landing by the UAV on the designated landing area is possible; and
if autonomous landing on the designated area is not possible, generating control commands configured to cause the UAV to descend in altitude; and

during the descent by the UAV:
identifying a second footprint based on the ground map, the second footprint including a second subset of the plurality of cells that include height statistics that satisfy the specified criterion, the second subset of the plurality of cells different than the subset of the plurality of cells;
designating a second landing area on the surface in the physical environment based on the identified second footprint; and

generating control commands configured to cause the UAV to autonomously maneuver to land on the designated second landing area.

10. The method of claim 1 , wherein a size and/or shape of the identified footprint is based on any of a size or shape of the UAV, a user preference, or a characteristic of a portion of the physical environment in proximity to the UAV.

11. The method of claim 1 , wherein the plurality of cells included in the ground map are rectangular in shape and are arranged as a two-dimensional (2D) rectangular grid that is continually updated to remain substantially centered on a position of the UAV while the UAV is in flight.

12. The method of claim 11 , wherein the 2D rectangular grid of the ground map is M cells wide by M cells long, and wherein the identified footprint includes the subset of the plurality of cells arranged in a rectangular shape of N cells wide by N cells long, wherein N is less than M.

13. The method of claim 1 , wherein identifying the footprint includes:
determining height statistics for a plurality of candidate footprints based on height statistics for cells included in each of the plurality of candidate footprints, wherein each of the plurality of candidate footprints includes a different subset of the plurality of cells of the ground map and is offset from each other by at least one cell; and
selecting the identified footprint from the plurality of candidate footprints that has a variance in height statistics that is below a maximum threshold and lower than the variance in height statistics of any of the other candidate footprints.

14. A method for autonomously landing an unmanned aerial vehicle (UAV), the UAV configured to autonomously track a subject in a physical environment;
monitoring, by a computer system, an operational status of the UAV while the UAV is in flight through the physical environment;
determining, by the computer system, that the operational status of the UAV does not satisfy an operational criterion based on the monitoring;
in response to determining that the operational status does not satisfy the operational criterion, automatically:
causing, by the computer system, the UAV to stop tracking the subject in the physical environment; and
causing, by the computer system, the UAV to perform an autonomous landing process by:
processing a ground map to identify a landing footprint, the ground map generated and continually updated based on images of the physical environment captured by an image capture device coupled to the UAV, the ground map including a plurality of cells, each of the plurality of cells including continually updated characteristic data based on the captured images, the landing footprint including a subset of the plurality of cells in the ground map that include characteristic data that satisfy a specified landing criterion;
designating a landing area on a surface in the physical environment based on the ident

Assignees

• Skydio Inc

Inventors

• Holtz Kristen Marie
• Martirosyan Hayk
• Zhu Jack Louis
• Bry Adam Parker
• Donahoe Matthew Joseph
• Bachrach Abraham Galton
• Henry Peter Benjamin
• Kennedy Ryan David

Classifications

• G06V20/17

  taken from planes or by drones · CPC title

• G06V20/58

  Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads · CPC title

• G06V10/225

  based on a marking or identifier characterising the area · CPC title

• G06V10/82

  using neural networks · CPC title

• G06V10/764

  using classification, e.g. of video objects · CPC title