Temporally consistent position estimation refinement for aerial refueling

Patent metadata

Field	Value
Publication number	US-2025191227-A1
Application number	US-202418949388-A
Country	US
Kind code	A1
Filing date	Nov 15, 2024
Priority date	Jan 5, 2022
Publication date	Jun 12, 2025
Grant date	—

Abstract

Aspects of the disclosure provide fuel receptacle position estimation for aerial refueling (derived from aircraft position estimation). A video stream comprising a plurality of video frames each showing an aircraft to be refueled, is received from a single camera. An initial position estimate is determined for the aircraft for the plurality of video frames, generating an estimated flight history for the aircraft. The estimated flight history for the aircraft is used to determine a temporally consistent refined position estimate, based on known aircraft flight path trajectories in an aerial refueling setting. The position of a fuel receptacle on the aircraft is determined, based on the refined position estimate for the aircraft, and an aerial refueling boom may be controlled to engage the fuel receptacle. Examples may use a deep learning neural network (NN) or optimization (e.g., bundle adjustment) to determine the refined position estimate from the estimated flight history.

First claim

What is claimed is:

1 . A method, comprising:
receiving a video stream showing an object to be refueled, wherein the video stream comprises a plurality of video frames; determining, for the plurality of video frames, an initial position estimate for the object, wherein the initial position estimate for the plurality of video frames comprises an estimated path history for the object; determining, based on at least the estimated path history for the object, a refined position estimate for the object; determining, based on at least the refined position estimate for the object, a position of a fuel receptacle on the object; determining a position of a refueling boom; tracking a distance between the refueling boom and the fuel receptacle; and controlling, based on at least the position of the fuel receptacle and the position of the boom, the refueling boom to engage the fuel receptacle.

2 . The method of claim 1 , further comprising:
generating an alert based on determining whether controlling the refueling boom to engage the fuel receptacle is within one or more safety parameters.

3 . The method of claim 1 , further comprising:
filtering the video stream and providing the filtered video stream to a feature extractor; outputting, based on the feature extractor, object features; and determining, based on the object features and using a two-dimensional (2D) to three-dimensional (3D) component, the initial position estimate for the object.

4 . The method of claim 1 , further comprising:
generating an overlay image comprising an object model projection that is based on the object and the refined position estimate,
wherein the overlay image further comprises a boom model projection that is based on at least the refueling boom and a boom tip position.

5 . The method of claim 1 , wherein the object is an aircraft, and
wherein the refined position estimate is determined further based on comparing the estimated path history with known flight trajectories for refueling the aircraft.

6 . The method of claim 1 , wherein the refined position estimate is determined via an estimate refiner that comprises a neural network (NN).

7 . The method of claim 1 , wherein the refueling boom comprises an extendable component.

8 . A device, comprising:
one or more memories; and one or more processors, communicatively coupled to the one or more memories, configured to:
receive a video stream showing an object to be refueled, wherein the video stream comprises a plurality of video frames;
determine, for the plurality of video frames, an initial position estimate for the object, wherein the initial position estimate for the plurality of video frames comprises an estimated path history for the object;
determine, based on at least the estimated path history for the object, refinement parameters, wherein the refinement parameters comprise a translation refinement and a rotational refinement;
determine, based on at least the estimated path history for the object and the refinement parameters, a refined position estimate for the object;
determine, based on at least the refined position estimate for the object, a position of a fuel receptacle on the object; and
control, based on at least the position of the fuel receptacle, a refueling boom to engage the fuel receptacle.

9 . The device of claim 8 , wherein the one or more processors are further configured to:
generate an alert based on determining whether controlling the refueling boom to engage the fuel receptacle is within one or more safety parameters.

10 . The device of claim 8 , wherein the object is an aircraft, and
wherein the refined position estimate is determined further based on comparing the estimated path history with known flight trajectories for refueling the aircraft.

11 . The device of claim 8 , wherein the one or more processors are further configured to:
filter the video stream and provide the filtered video stream to a feature extractor; output, based on the feature extractor, object features; and determine, based on the object features and using a two-dimensional (2D) to three-dimensional (3D) component, the initial position estimate for the object.

12 . The device of claim 8 , wherein the one or more processors are further configured to:
generate an overlay image comprising an object model projection that is based on the object and the refined position estimate,
wherein the overlay image further comprises a boom model projection that is based on at least the refueling boom and a boom tip position.

13 . The device of claim 8 , wherein the refueling boom comprises an extendable component.

14 . The device of claim 8 , wherein the refined position estimate is determined via an estimate refiner that comprises a neural network.

15 . A non-transitory computer-readable medium storing a set of instructions, the set of instructions comprising:
one or more instructions that, when executed by one or more processors of a device, cause the device to:
receive a video stream showing an object to be refueled, wherein the video stream comprises a plurality of video frames;
determine, for the plurality of video frames, an initial position estimate for the object, wherein the initial position estimate for the plurality of video frames comprises an estimated path history for the object;
determine, based on at least the estimated path history for the object, refinement parameters, wherein the refinement parameters comprise a translation refinement and a rotational refinement;
determine, based on at least the estimated path history for the object and the refinement parameters, a refined position estimate for the object;
determine, based on at least the refined position estimate for the object, a position of a fuel receptacle on the object; and
control, based on at least the position of the fuel receptacle, a refueling boom to engage the fuel receptacle.

16 . The non-transitory computer-readable medium of claim 15 , wherein the one or more instructions further cause the device to:
generate an alert based on determining whether controlling the refueling boom to engage the fuel receptacle is within one or more safety parameters.

17 . The non-transitory computer-readable medium of claim 15 , wherein the object is an aircraft, and
wherein the refined position estimate is determined further based on comparing the estimated path history with known flight trajectories for refueling the aircraft.

18 . The non-transitory computer-readable medium of claim 15 , wherein the one or more instructions further cause the device to:
filter the video stream and provide the filtered video stream to a feature extractor; output, based on the feature extractor, object features; and determine, based on the object features and using a two-dimensional (2D) to three-dimensional (3D) component, the initial position estimate for the object.

19 . The non-transitory computer-readable medium of claim 15 , wherein the one or more processors are further configured to:
generate an overlay image comprising an object model projection that is based on the object and the refined position estimate,
wherein the overlay image further comprises a boom model projection that is based on at least the refueling boom and a boom tip position.

20 . The non-transitory computer-readable medium of claim 15 , wherein the refined position estimate is determined via an estimate refiner that comprises a neural network.

Assignees

• Boeing Co

Inventors

• Nguyen Leon Nhat
• Smith Haden Harrison
• Hung Fan Hin
• Khosla Deepak

Classifications

• G05D1/695

  for maintaining a fixed relative position of the vehicles, e.g. for convoy travelling or formation flight · CPC title

• G06T19/006

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• B64D39/00 Primary

  Refuelling during flight · CPC title

• G05D1/104

  involving a plurality of aircrafts, e.g. formation flying (traffic control systems for aircraft G08G5/00) · CPC title

• G03B15/006

  Apparatus mounted on flying objects · CPC title