Systems and methods for calculating refueling tanker boom tip 3D position for aerial refueling

Patent metadata

Field	Value
Publication number	US-12493979-B2
Application number	US-202318314681-A
Country	US
Kind code	B2
Filing date	May 9, 2023
Priority date	May 9, 2023
Publication date	Dec 9, 2025
Grant date	Dec 9, 2025

Abstract

Disclosed herein is methods, systems, and aircraft for performing image analysis for aiding refueling operations. A tanker aircraft includes a single camera, a refueling boom, a camera configured to generate a 2D image of the refueling boom, a processor, and non-transitory computer readable storage media storing code. The code being executable by the processor to perform operations comprising receiving the 2D image from the single camera, determining 2D keypoints of the refueling boom located within the 2D image based on a predefined point model of the refueling boom, determining keypoints in 3D space based on the 2D keypoints to produce 3D keypoints, determining a 6DOF pose using the 2D keypoints and the 3D keypoints, and estimating a position of a tip of the refueling boom based on the 6DOF pose.

First claim

What is claimed is:

1 . A method comprising:
receiving a two-dimensional (2D) image from a single camera of a tanker aircraft; determining 2D keypoints of a refueling boom located within the 2D image based on a predefined point model of the refueling boom, the 2D keypoints comprising a 2D boom tip keypoint within the 2D image; determining a 6 degree-of-freedom (6DOF) pose using the 2D keypoints and the corresponding three-dimensional (3D) space 3D keypoints; identifying a horizontal plane based on the 2D boom tip keypoint; generating a raytracing value of the tip of the refueling boom by identifying an intersection of a longitudinal boom axis and the horizontal plane; and estimating a position of a tip of the refueling boom based on the 6DOF pose and on the generated raytracing value.

2 . The method of claim 1 , wherein determining the 6DOF pose comprises performing a perspective-n-point algorithm using the 2D keypoints and the 3D keypoints to produce the 6DOF pose.

3 . The method of claim 1 , wherein estimating the position of the tip of the refueling boom comprises:
receiving a boom tip extension value; and estimating the position of the tip of the refueling boom based on the boom tip extension value.

4 . The method of claim 3 , wherein receiving the boom tip extension value comprises receiving the boom tip extension value from a sensor.

5 . The method of claim 1 , wherein:
generating the raytracing value comprises:
identifying the 2D boom tip key point within the 2D image; and
producing a 3D boom tip key point based on the 2D boom tip key point; and
the horizontal plane is between the camera and the 3D boom tip key point.

6 . The method of claim 5 , wherein identifying the horizontal plane comprises parameterizing the horizontal plane based on the 3D boom tip key point.

7 . The method of claim 1 , further comprising sending the position of the tip of the refueling boom to an output device.

8 . The method of claim 7 , wherein the output device comprises an automated refueling system, a pilot director light system, or a boom operator interface.

9 . The method of claim 1 , wherein identifying the intersection of the longitudinal boom axis and the horizontal plane comprises determining a slope of the horizontal plane based on the 2D keypoints.

10 . A tanker aircraft comprising:
a single camera; a refueling boom; a camera configured to generate a two-dimensional (2D) image of the refueling boom; a processor; and non-transitory computer readable storage media storing code, the code being executable by the processor to perform operations comprising:
receiving the two-dimensional (2D) image from the single camera;
determining 2D keypoints of the refueling boom located within the 2D image based on a predefined point model of the refueling boom, the 2D keypoints comprising a 2D boom tip keypoint within the 2D image;
determining a 6 degree-of-freedom (6DOF) pose using the 2D keypoints and the corresponding three-dimensional (3D) space 3D keypoints;
identifying a horizontal plane based on the 2D boom tip keypoint;
generating a raytracing value of the tip of the refueling boom by identifying an intersection of a longitudinal boom axis and the horizontal plane; and
estimating a position of a tip of the refueling boom based on the 6DOF pose and on the generated raytracing value.

11 . The tanker aircraft of claim 10 , wherein transforming the 2D keypoints to 3D space comprises performing a perspective-n-point algorithm using the 2D keypoints and the 3D keypoints to produce the 6DOF pose.

12 . The tanker aircraft of claim 10 , wherein:
the tanker aircraft further comprises a boom tip sensor; and estimating the position of the tip of the refueling boom comprises:
receiving a boom tip extension value from the boom tip sensor; and
estimating the position of the tip of the refueling boom based on the boom tip extension value.

13 . The tanker aircraft of claim 10 , wherein generating the raytracing value comprises identifying the 2D boom tip key point within the 2D image to produce a 3D boom tip key point and the horizontal plane is between the camera and the 3D boom tip key point.

14 . The tanker aircraft of claim 10 , wherein:
the tanker aircraft further comprises an output device; and the processor further performs an operation of sending the position of the tip of the refueling boom to the output device.

15 . The tanker aircraft of claim 14 , wherein the output device comprises an automated refueling system, a pilot director light system, or a boom operator interface.

16 . A refueling system comprising:
a processor; and non-transitory computer readable storage media storing code, the code being executable by the processor to perform operations comprising:
receiving a two-dimensional (2D) image from a single camera of a tanker aircraft;
determining 2D keypoints of a refueling boom located within the 2D image based on a predefined point model of the refueling boom, the 2D keypoints comprising a 2D boom tip keypoint within the 2D image;
determining a 6 degree-of-freedom (6DOF) pose using the 2D keypoints and the corresponding three-dimensional (3D) space 3D keypoints;
identifying a horizontal plane based on the 2D boom tip keypoint;
generating a raytracing value of the tip of the refueling boom by identifying an intersection of a longitudinal boom axis and the horizontal plane; and
estimating a position of a tip of the refueling boom based on the 6DOF pose and on the generated raytracing value.

17 . The refueling system of claim 16 , wherein transforming the 2D keypoints to 3D space comprises performing a perspective-n-point algorithm using the 2D keypoints and 3D keypoints to produce the 6DOF pose.

18 . The refueling system of claim 16 , wherein estimating the position of the tip of the refueling boom comprises:
receiving a boom tip extension value from a boom tip sensor; and estimating the position of the tip of the refueling boom based on the boom tip extension value.

19 . The refueling system of claim 16 , wherein:
generating the raytracing value comprises identifying the 2D boom tip key point within the 2D image to produce a 3D boom tip keypoint, and the horizontal plane between the camera and the 3D boom tip key point.

20 . The refueling system of claim 16 , wherein the processor further performs an operation of sending the position of the tip of the refueling boom to an automated refueling system, a pilot director light system, or a boom operator interface.

Assignees

• Boeing Co

Inventors

• Smith Haden
• Khosla Deepak
• Nguyen Leon
• Hung Fan Hin
• Hatcher Justin
• O'Shea Daniel
• Chamberlain Shawn
• Santiago Luis-Alberto
• Jang Jung Soon

Classifications

• G06T15/06

  Ray-tracing · CPC title

• G06T7/73 Primary

  using feature-based methods · CPC title

• B64D39/00

  Refuelling during flight · CPC title

• G06T7/75

  involving models · CPC title

• G06N3/08

  Learning methods · CPC title