Semantic abort of unmanned aerial vehicle deliveries

Patent metadata

Field	Value
Publication number	US-12283099-B2
Application number	US-202217657558-A
Country	US
Kind code	B2
Filing date	Mar 31, 2022
Priority date	Mar 31, 2022
Publication date	Apr 22, 2025
Grant date	Apr 22, 2025

Abstract

A method includes capturing, by a sensor on an unmanned aerial vehicle (UAV), an image of a delivery location. The method also includes determining, based on the image of the delivery location, a segmentation image. The segmentation image segments the delivery location into a plurality of pixel areas with corresponding semantic classifications. The method additionally includes determining, based on the segmentation image, a percentage of obstacle pixels within a surrounding area of a delivery point at the delivery location, wherein each obstacle pixel has a semantic classification indicative of an obstacle in the delivery location. The method further includes based on the percentage of obstacle pixels being above a threshold percentage, aborting a delivery process of the UAV.

First claim

What is claimed is:

1. A method comprising:
capturing, by a sensor on an unmanned aerial vehicle (UAV), an image of a delivery location;
determining, based on the image of the delivery location, a segmentation image, wherein the segmentation image segments the delivery location into a plurality of pixel areas with corresponding semantic classifications;
determining, based on the segmentation image, a percentage of obstacle pixels within a surrounding area of a delivery point at the delivery location, wherein each obstacle pixel has a semantic classification indicative of an obstacle in the delivery location; and
based on the percentage of obstacle pixels being above a threshold percentage, aborting a delivery process of the UAV.

2. The method of claim 1 , further comprising:
determining, by a delivery point selection module, the delivery point at the delivery location; and

navigating, by the UAV, to the delivery point, wherein aborting the delivery process of the UAV is performed after determining the delivery point at the delivery location and after navigating to the delivery point.

3. The method of claim 2 , wherein determining the delivery point at the delivery location comprises:
determining an initial delivery point;

determining that the initial delivery point is within a threshold distance of an obstacle; and
selecting an adjusted delivery point which is at least the threshold distance away from the initial delivery point, wherein the adjusted delivery point is the delivery point at the delivery location.

4. The method of claim 1 , wherein the method is carried out during a descent process of the UAV, wherein aborting the delivery process of the UAV is aborting the descent process of the UAV.

5. The method of claim 1 , wherein the method further comprises:
capturing one or more additional images of the delivery location;
determining, based on the one or more additional images of the delivery location, one or more additional segmentation images;
determining, based on the one or more additional segmentation images, one or more additional percentages of obstacle pixels within a surrounding area of a delivery point at the delivery location;
determining a total count of the one or more percentages of additional obstacle pixels and the percentage of obstacle pixels being above a threshold percentage; and
determining that the total count is greater than a threshold count, wherein aborting the delivery process of the UAV is further based on determining that the total count is greater than the threshold count.

6. The method of claim 1 , wherein the method further comprises:
capturing one or more additional images of the delivery location;
determining, based on the one or more additional images of the delivery location, one or more additional segmentation images;
determining, based on the one or more additional segmentation images, one or more additional percentages of obstacle pixels within a surrounding area of a delivery point at the delivery location;
determining an average percentage of the one or more additional percentages of obstacle pixels and the percentage of obstacle pixels; and
determining that the average percentage is greater than a threshold average percentage, wherein aborting the delivery process of the UAV is further based on determining that the average percentage is greater than the threshold average percentage.

7. The method of claim 1 , wherein the method further comprises:
capturing one or more additional images of the delivery location;
determining, based on the one or more additional images of the delivery location, one or more additional segmentation images;
determining, based on the one or more additional segmentation images, one or more additional percentages of obstacle pixels within a surrounding area of a delivery point at the delivery location; and
determining an uncertainty measure based on the one or more additional percentages of obstacle pixels and the percentage of obstacle pixels, wherein aborting the delivery process of the UAV is further based on determining that the uncertainty measure is greater than a threshold uncertainty measure.

8. The method of claim 1 , wherein the method further comprises:
capturing one or more additional images of the delivery location at a set time interval, wherein aborting the delivery process of the UAV is further based on the additional one or more images.

9. The method of claim 1 , wherein the method further comprises:
descending the UAV to a predetermined altitude above the delivery location, wherein capturing the image of the delivery location is performed after descending to the predetermined altitude.

10. The method of claim 1 , wherein the delivery point is a point over which the UAV is hovering when capturing the image of the delivery location.

11. The method of claim 1 , wherein capturing the image of the delivery location comprises capturing a tilted image of the delivery location, wherein determining a percentage of obstacle pixels within a surrounding area of a delivery point at the delivery location is based on the tilted image of the delivery location.

12. The method of claim 1 , wherein the image of the delivery location is a 2-dimensional (2D) image, wherein the method further comprises:
determining, based on the 2D image, a 3-dimensional (3D) depth image of the delivery location, wherein determining the percentage of obstacle pixels within the surrounding area of the delivery point at the delivery location is further based on the 3D depth image.

13. The method of claim 1 , wherein the image of the delivery location is a 2D image, wherein determining the segmentation image is based on applying a pre-trained machine learning model to the image.

14. The method of claim 1 , wherein the delivery process is to deliver a payload with one or more dimensions, wherein method further comprises:
determining the threshold percentage based on the one or more dimensions of the payload.

15. The method of claim 1 , wherein the sensor on the UAV faces downward, and wherein the image of the delivery location captured by the sensor is representative of the delivery location below the UAV.

16. The method of claim 1 , wherein the semantic classifications are selected from a predetermined set of semantic classifications, wherein the predetermined set of semantic classifications includes at least semantic classifications corresponding to vegetation, building, and road.

17. An unmanned aerial vehicle (UAV), comprising:
a sensor; and
a control system configured to:
capture, by a sensor on an unmanned aerial vehicle (UAV), an image of a delivery location;
determine, based on the image of the delivery location, a segmentation image, wherein the segmentation image segments the delivery location into a plurality of pixel areas with corresponding semantic classifications;
determine, based on the segmentation image, a percentage of obstacle pixels within a surrounding area of a delivery point at the delivery location, wherein each obstacle pixel has a semantic classification indicative of an obstacle in the delivery location; and
based on the percentage of obstacle pixels being above a threshold percentage, abort a delivery process of the UAV.

18. The UAV of claim 17 , wherein the sensor is a downward facing camera attached to the UAV, wherein the image of the delivery location captured by the downward facing camera is representative of the delivery location below the UAV.

19. The UAV of claim 17 , further comprising a d

Assignees

• Wing Aviation Llc

Inventors

• Shoeb Ali
• Hammond Marcus
• Cobar Christopher
• Krafka Kyle
• Julian Kyle
• Jenkins Kevin

Classifications

• G05D1/689

  Pointing payloads towards fixed or moving targets (positioning towed, pushed or suspended implements G05D1/672) · CPC title

• G05D1/485

  Control of rate of change of altitude or depth · CPC title

• G06T2207/30261

  Obstacle · CPC title

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title

• G06T2207/20081

  Training; Learning · CPC title