Visual detection and localization of package autoloaders by UAV

Patent metadata

Field	Value
Publication number	US-12412301-B2
Application number	US-202318211919-A
Country	US
Kind code	B2
Filing date	Jun 20, 2023
Priority date	Jun 20, 2023
Publication date	Sep 9, 2025
Grant date	Sep 9, 2025

Abstract

A technique for a UAV includes: acquiring an aerial image of an area below a UAV that includes one or more instances of an object; analyzing the aerial image with an image classifier to classify select pixels of the aerial image as being keypoint pixels associated with keypoints of the object; grouping the keypoint pixels into one or more groups each associated with one of the instances of the object, wherein first keypoint pixels of the keypoint pixels are grouped into a first group of the one or more groups associated with a first instance of the one or more instances of the object; generating an estimate of a relative position of the UAV to the first instance of the object based at least upon a machine vision analysis of the first keypoint pixels; and navigating the UAV into alignment with the first instance based upon the estimate.

First claim

What is claimed is:

1. A method of operation of an unmanned aerial vehicle (UAV), the method comprising:
acquiring an aerial image with an onboard camera of the UAV, wherein the aerial image includes one or more instances of an object;
analyzing the aerial image with an image classifier to classify select pixels of the aerial image as being keypoint pixels associated with keypoints of the object;
grouping the keypoint pixels into one or more groups including a first group of the keypoint pixels, wherein each of the one or more groups of the keypoint pixels is associated with a corresponding one of the one or more instances of the object, and wherein the first group of the keypoint pixels is associated with a first instance of the one or more instances of the object;
generating an estimate of a relative position of the UAV to the first instance of the object based at least partially upon a machine vision analysis of the first group of the keypoint pixels;
acquiring state information of the UAV from one or more onboard sensors of the UAV;
performing a consistency check on the estimate of the relative position by referencing the state information and at least one assumption of an orientation of the first instance of the object;
rejecting the estimate of the relative position as an erroneous estimate if the consistency check fails; and
navigating the UAV relative to the first instance of the object based upon the estimate of the relative position if the consistency check passes,
wherein the object comprises an apparatus adapted to transfer a package to the UAV.

2. The method of claim 1 , wherein the object comprises an autoloading apparatus that is adapted to load the package onto a line deployed from the UAV and the keypoints comprise visually distinctive locations on the autoloading apparatus.

3. The method of claim 2 , further comprising:
comparing the first group of the keypoint pixels to a software model of the autoloading apparatus that specifies a spatial relationship of the keypoints on the autoloading apparatus; and
rejecting the first group of the keypoint pixels as a false detection of the autoloading apparatus when the spatial relationship of the first group of the keypoint pixels fails to match the spatial relationship of the keypoints defined in the software model within a threshold error.

4. The method of claim 1 , wherein the machine vision analysis includes a perspective-n-point analysis of the first group of the keypoint pixels.

5. The method of claim 1 , wherein grouping the keypoint pixels comprises:
analyzing the aerial image to detect the one or more instances of the object in the aerial image;
generating one or more bounding boxes that each encircles an associated one of the one or more instances of the object detected; and
grouping the keypoint pixels that fall within a common one of the bounding boxes.

6. The method of claim 5 , further comprising:
rejecting any one of the keypoint pixels that does not fall within one of the bounding boxes as a false keypoint detection.

7. The method of claim 6 , wherein the one or more bounding boxes and the keypoint pixels are independent outputs of the image classifier, and wherein the image classifier comprises a neural network trained to separately identify both the keypoints and the object.

8. The method of claim 1 , wherein generating the estimate of the relative position of the UAV to the first instance of the object comprises:
generating the estimate of the relative position in a world frame coordinate system aligned to a fixed direction.

9. The method of claim 1 , further comprising:
tracking each of the one or more instances of the object in the aerial image between acquisitions of sequential aerial images including the aerial image by categorizing each detected instance as belonging to an existing object track or a new object track, wherein the categorizing is determined based on a displacement threshold determined from sensor readings output from one or more onboard motion sensors of the UAV, wherein the sensor readings are acquired between the acquisitions of the sequential aerial images.

10. The method of claim 1 , wherein analyzing the aerial image with the image classifier to classify the select pixels of the aerial image as being keypoint pixels comprises classifying the select pixels themselves or classifying locations within the aerial image associated with the select pixels.

11. At least one non-transitory computer-readable medium storing instructions that, when executed by a control system of an unmanned aerial vehicle (UAV), will cause the UAV to perform operations comprising:
acquiring an aerial image with an onboard camera of the UAV of an area below the UAV that includes one or more instances of an autoloader that is adapted to load a package onto a line deployed from the UAV;
analyzing the aerial image with an image classifier to classify select pixels of the aerial image as being keypoint pixels associated with keypoints of the autoloader;
grouping the keypoint pixels into one or more groups including a first group of the keypoint pixels, wherein each of the one or more groups of the keypoint pixels is associated with a corresponding one of the one or more instances of the autoloader, and wherein the first group of the keypoint pixels is associated with a first instance of the one or more instances of the autoloader;
generating an estimate of a relative position of the UAV to the first instance of the object based at least partially upon a machine vision analysis of the first group of the keypoint pixels;
comparing the first group of the keypoint pixels to a software model of the autoloader that specifies a spatial relationship of the keypoints on the autoloader;
rejecting the first group of the keypoint pixels as a false detection of the autoloader if the spatial relationship of the first group of the keypoint pixels fails to match the spatial relationship of the keypoints defined in the software model within a threshold error; and
navigating the UAV relative to the first instance of the autoloader based upon the estimate of the relative position if the first group of the keypoint pixels are not rejected as the false detection.

12. The at least one non-transitory computer-readable medium of claim 11 , wherein the keypoints comprise visually distinctive locations on the autoloader.

13. The at least one non-transitory computer-readable medium of claim 11 , wherein the machine vision analysis includes a perspective-n-point analysis of the first group of the keypoint pixels.

14. The at least one non-transitory computer-readable medium of claim 11 , wherein grouping the keypoint pixels comprises:
analyzing the aerial image to detect the one or more instances of the autoloader in the aerial image;
generating one or more bounding boxes that each encircles an associated one of the one or more instances of the autoloader detected; and
grouping the keypoint pixels that fall within a common one of the bounding boxes.

15. The at least one non-transitory computer-readable medium of claim 14 , further storing instructions that, when executed by the control system, will cause the UAV to perform further operations, comprising:
rejecting any one of the keypoint pixels that does not fall within one of the bounding boxes as being a false keypoint detection.

16. The at least one non-transitory computer-readable medium of claim 15 , wherein the one or more bounding boxes and the keypoint pixels are independent outputs of the image classifier, and wherein the image classifier comprises a neural network

Assignees

• Wing Aviation Llc

Inventors

• Stonestrom David
• Shoeb Ali
• Hammond Marcus

Classifications

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title

• G06T2207/10032

  Satellite or aerial image; Remote sensing · CPC title

• B64D45/00

  Aircraft indicators or protectors not otherwise provided for · CPC title

• B64U2101/66

  for retrieving parcels · CPC title

• B64U10/20

  Vertical take-off and landing [VTOL] aircraft (flying platforms B64U10/13; helicopters B64U10/17) · CPC title