Determining a current pose estimate of an aircraft relative to a runway to support the aircraft on approach

Patent metadata

Field	Value
Publication number	US-12148183-B2
Application number	US-202117527783-A
Country	US
Kind code	B2
Filing date	Nov 16, 2021
Priority date	Dec 18, 2020
Publication date	Nov 19, 2024
Grant date	Nov 19, 2024

Abstract

A method is provided for supporting an aircraft approaching a runway on an airfield. The method includes receiving a sequence of images of the airfield, captured by a camera onboard the aircraft approaching the runway. For at least one image of the sequence of images, the method includes applying the image(s) to a machine learning model trained to predict a pose of the aircraft relative to the runway. The machine learning model is configured to map the image(s) to the pose based on a training set of labeled images with respective ground truth poses of the aircraft relative to the runway. The pose is output as a current pose estimate of the aircraft relative to the runway for use in at least one of monitoring the current pose estimate, generating an alert based on the current pose estimate, or guidance or control of the aircraft on a final approach.

First claim

What is claimed is:

1. An apparatus for supporting an aircraft approaching a runway on an airfield, the apparatus comprising:
a memory configured to store computer-readable program code; and
processing circuitry configured to access the memory, and execute the computer-readable program code to cause the apparatus to at least:
receive a sequence of images of the airfield, captured by at least one camera onboard the aircraft approaching the runway; and for at least one image of the sequence of images,
apply the at least one image to a machine learning model trained to predict a pose of the aircraft relative to the runway, the machine learning model configured to map the at least one image to the pose based on a training set of labeled images with respective ground truth poses of the aircraft relative to the runway; and
output the pose as a current pose estimate of the aircraft relative to the runway for use in at least one of monitoring the current pose estimate, generating an alert based on the current pose estimate, or guidance or control of the aircraft on a final approach.

2. The apparatus of claim 1 , wherein the apparatus caused to apply the at least one image to the machine learning model includes the apparatus caused to at least:
apply the at least one image to the machine learning model trained to predict a pose of the at least one camera in camera coordinates; and
transform the camera coordinates for the at least one camera to corresponding runway-framed local coordinates and thereby predict the pose of the aircraft relative to the runway.

3. The apparatus of claim 1 , wherein the at least one image and the labeled images are in a non-visible light spectrum.

4. The apparatus of claim 1 , wherein the labeled images are mono-channel images, the at least one image is a multi-channel image, and the processing circuitry is configured to execute the computer-readable program code to cause the apparatus to further convert the multi-channel image to a mono-channel image that is applied to the machine learning model.

5. The apparatus of claim 1 , wherein the processing circuitry is configured to execute the computer-readable program code to cause the apparatus to further crop the at least one image to reduce a field of view of the at least one camera, and magnify only a portion of the at least one image on which the runway is located, before the at least one image is applied to the machine learning model.

6. The apparatus of claim 1 , wherein the processing circuitry is configured to execute the computer-readable program code to cause the apparatus to further generate the training set of labeled images, including the apparatus caused to at least:
receive earlier images of the airfield, captured by the at least one camera onboard the aircraft or a second aircraft approaching the runway, and the respective ground truth poses of the aircraft or the second aircraft relative to the runway; and
label the earlier images with the respective ground truth poses of the aircraft to generate the training set of labeled images.

7. The apparatus of claim 1 , wherein the processing circuitry is configured to execute the computer-readable program code to cause the apparatus to further generate the training set of labeled images, including the apparatus caused to at least:
execute a flight simulator configured to artificially re-create flight of the aircraft approaching the runway on the airfield;
capture synthetic images of the airfield, and determine the respective ground truth poses of the aircraft relative to the runway, from the flight simulator; and
label the synthetic images with the respective ground truth poses of the aircraft to generate the training set of labeled images.

8. The apparatus of claim 1 , wherein the apparatus caused to apply the at least one image to the machine learning model includes the apparatus caused to apply the at least one image to machine learning models trained to predict respective components of the pose of the aircraft relative to the runway, the machine learning models configured to determine values of the components and thereby the pose of the aircraft relative to the runway.

9. The apparatus of claim 1 , wherein the apparatus caused to apply the at least one image to the machine learning model includes the apparatus caused to apply the at least one image to machine learning models trained to predict multiple current pose estimates according to different algorithms, and the processing circuitry is configured to execute the computer-readable program code to cause the apparatus to further at least:
determine confidence intervals associated with respective ones of the multiple current pose estimates; and
perform a sensor fusion of the multiple current pose estimates using the confidence intervals to determine the current pose estimate of the aircraft relative to the runway.

10. The apparatus of claim 1 , wherein the at least one camera is located onboard the aircraft in a configuration that allows the at least one camera to capture a view of the environment ahead of the aircraft in a direction of travel of the aircraft.

11. A method of supporting an aircraft approaching a runway on an airfield, the method comprising:
receiving a sequence of images of the airfield, captured by at least one camera onboard the aircraft approaching the runway; and for at least one image of the sequence of images,
applying the at least one image to a machine learning model trained to predict a pose of the aircraft relative to the runway, the machine learning model configured to map the at least one image to the pose based on a training set of labeled images with respective ground truth poses of the aircraft relative to the runway; and
outputting the pose as a current pose estimate of the aircraft relative to the runway for use in at least one of monitoring the current pose estimate, generating an alert based on the current pose estimate, or guidance or control of the aircraft on a final approach.

12. The method of claim 11 , wherein applying the at least one image to the machine learning model includes:
applying the at least one image to the machine learning model trained to predict a pose of the at least one camera in camera coordinates; and
transforming the camera coordinates for the at least one camera to corresponding runway-framed local coordinates and thereby predict the pose of the aircraft relative to the runway.

13. The method of claim 11 , wherein the at least one image and the labeled images are in a non-visible light spectrum.

14. The method of claim 11 , wherein the labeled images are mono-channel images, the at least one image is a multi-channel image, and the method further comprises converting the multi-channel image to a mono-channel image that is applied to the machine learning model.

15. The method of claim 11 further comprising cropping the at least one image to reduce a field of view of the at least one camera, and magnifying only a portion of the at least one image on which the runway is located, before the at least one image is applied to the machine learning model.

16. The method of claim 11 further comprising generating the training set of labeled images, including at least:
receiving earlier images of the airfield, captured by the at least one camera onboard the aircraft or a second aircraft approaching the runway, and the respective ground truth poses of the aircraft or the second aircraft relative to the runway; and
labeling the earlier images with the respective ground truth poses of the aircraft to generate the training set of labeled images.

Assignees

• Boeing Co

Inventors

• Evans Nick S
• Muir Eric R
• Billhartz Cullen W
• Corleis Jasper P

Classifications

• G08G5/54

  for approach or landing · CPC title

• G08G5/74

  for monitoring terrain · CPC title

• G08G5/21 Primary

  located onboard the aircraft · CPC title

• G06T2207/10032

  Satellite or aerial image; Remote sensing · CPC title

• G06T2207/20164

  Salient point detection; Corner detection · CPC title