Pixel-by-pixel segmentation of aerial imagery for autonomous vehicle control

Patent metadata

Field	Value
Publication number	US-12242282-B2
Application number	US-202217946972-A
Country	US
Kind code	B2
Filing date	Sep 16, 2022
Priority date	Sep 16, 2022
Publication date	Mar 4, 2025
Grant date	Mar 4, 2025

Abstract

In some embodiments, an unmanned aerial vehicle (UAV) is provided. The UAV comprises one or more processors; a camera; one or more propulsion devices; and a computer-readable medium having instructions stored thereon that, in response to execution by the one or more processors, cause the UAV to perform actions comprising: receiving at least one image captured by the camera; generating labels for pixels of the at least one image by providing the at least one image as input to a machine learning model; identifying one or more landing spaces in the at least one image based on the labels; determining a relative position of the UAV with respect to the one or more landing spaces; and transmitting signals to the one or more propulsion devices based on the relative position of the UAV with respect to the one or more landing spaces.

First claim

What is claimed is:

1. A non-transitory computer-readable medium having computer-executable instructions stored thereon that, in response to execution by one or more processors of an unmanned aerial vehicle (UAV), cause the UAV to perform actions comprising:
capturing at least one image using a camera of the UAV;
generating labels for pixels of the at least one image by providing the at least one image as input to a machine learning model, wherein the labels include an unoccupied landing space pixel label, an occupied landing space pixel label, and a non-landing space pixel label;
identifying one or more landing spaces in the at least one image based on the labels;
determining a relative position of the UAV with respect to the one or more landing spaces; and
transmitting signals to one or more propulsion devices of the UAV based on the relative position of the UAV with respect to the one or more landing spaces, wherein the signals cause the UAV to autonomously execute a maneuver, and wherein the maneuver is a stationary hover or a landing in an unoccupied landing space.

2. The non-transitory computer-readable medium of claim 1 , wherein the maneuver is the landing in the unoccupied landing space, and wherein the actions further comprise:
identifying the unoccupied landing space from the one or more landing spaces as an endpoint of a navigation path; and
determining control signals for causing the UAV to transit the navigation path and land at the unoccupied landing space;
wherein transmitting the signals to the one or more propulsion devices of the UAV includes transmitting the determined control signals.

3. The non-transitory computer-readable medium of claim 1 , wherein the maneuver is the stationary hover, and wherein the actions further comprise:
determining control signals for causing the UAV to hover in a constant position in relation to the position of one or more landing spaces; and
wherein transmitting the signals to the one or more propulsion devices of the UAV includes transmitting the determined control signals.

4. The non-transitory computer-readable medium of claim 1 , wherein determining the position of one or more landing spaces based on the labels for the pixels of the at least one image includes identifying at least one group of pixels having unoccupied landing space pixel labels or occupied landing space pixel labels as being landing spaces based on one or more heuristics.

5. The non-transitory computer-readable medium of claim 4 , wherein the one or more heuristics include one or more of meeting a minimum size threshold, having a predetermined shape, and having a minimum number of pixels with occupied landing space pixel labels over a predetermined amount of time.

6. The non-transitory computer-readable medium of claim 1 , wherein generating labels for pixels of the at least one image further includes providing one or more of a heading, a pitch, a yaw, and an altitude of the UAV as additional input to the machine learning model.

7. The non-transitory computer-readable medium of claim 1 , wherein the actions further comprise determining an altitude of the UAV based on the position of the one or more landing spaces.

8. The non-transitory computer-readable medium of claim 1 , wherein providing the at least one image as input to a machine learning model includes providing a plurality of images as input to the machine learning model that include an image captured at a time t and one or more images captured before time t in order to generate labels for pixels in the image captured at the time t.

9. The non-transitory computer-readable medium of claim 1 , wherein the machine learning model is an encoder-decoder machine learning model.

10. The non-transitory computer-readable medium of claim 1 , wherein each landing space is a charging pad.

11. An unmanned aerial vehicle (UAV), comprising:
one or more processors;
a camera;
one or more propulsion devices; and
a non-transitory computer-readable medium having computer-executable instructions stored thereon that, in response to execution by the one or more processors, cause the UAV to perform actions comprising:
receiving at least one image captured by the camera;
generating labels for pixels of the at least one image by providing the at least one image as input to a machine learning model, wherein the labels include an unoccupied landing space pixel label, an occupied landing space pixel label, and a non-landing space pixel label;
identifying one or more landing spaces in the at least one image based on the labels;
determining a relative position of the UAV with respect to the one or more landing spaces; and
transmitting signals to the one or more propulsion devices based on the relative position of the UAV with respect to the one or more landing spaces.

12. The UAV of claim 11 , wherein the actions further comprise:
identifying an unoccupied landing space of the one or more landing spaces as an endpoint of a navigation path; and
determining control signals for causing the UAV to transit the navigation path and land at the unoccupied landing space;
wherein transmitting the signals to the one or more propulsion devices of the UAV includes transmitting the determined control signals.

13. The UAV of claim 11 , wherein the actions further comprise:
determining control signals for causing the UAV to hover in a constant position in relation to the position of one or more landing spaces;
wherein transmitting the signals to the one or more propulsion devices of the UAV includes transmitting the determined control signals.

14. The UAV of claim 11 , wherein determining the position of one or more landing spaces based on the labels for the pixels of the at least one image includes identifying at least one group of pixels having unoccupied landing space pixel labels or occupied landing space pixel labels as being landing spaces based on one or more heuristics.

15. The UAV of claim 14 , wherein the one or more heuristics include one or more of meeting a minimum size threshold, having a predetermined shape, and having a minimum number of pixels with occupied landing space pixel labels over a predetermined amount of time.

16. The UAV of claim 11 , wherein generating labels for pixels of the at least one image further includes providing one or more of a heading, a pitch, a yaw, and an altitude of the UAV as additional input to the machine learning model.

17. The UAV of claim 11 , wherein the actions further comprise determining an altitude of the UAV based on the position of the one or more landing spaces.

18. The UAV of claim 11 , wherein providing the at least one image as input to a machine learning model includes providing a plurality of images as input to the machine learning model that include an image captured at a time t and one or more images captured before time t in order to generate labels for pixels in the image captured at the time t.

19. The UAV of claim 11 , wherein the machine learning model is an encoder-decoder machine learning model.

20. The UAV of claim 11 , wherein each landing space is a charging pad.

Assignees

• Wing Aviation Llc

Inventors

• Cobar Christopher
• Gabor Jeremie
• Shoeb Ali

Classifications

• G05D1/654

  Landing (docking at a base station G05D1/661) · CPC title

• G05D1/49

  Control of attitude, i.e. control of roll, pitch or yaw · CPC title

• G05D2109/20

  Aircraft, e.g. drones · CPC title

• B64C39/024

  of the remote controlled vehicle type, i.e. RPV · CPC title

• B64U2101/30

  for imaging, photography or videography · CPC title