Temporally consistent position estimation refinement for aerial refueling

Patent metadata

Field	Value
Publication number	US-12148184-B2
Application number	US-202217569448-A
Country	US
Kind code	B2
Filing date	Jan 5, 2022
Priority date	Jan 5, 2022
Publication date	Nov 19, 2024
Grant date	Nov 19, 2024

Abstract

Aspects of the disclosure provide fuel receptacle position estimation for aerial refueling (derived from aircraft position estimation). A video stream comprising a plurality of video frames each showing an aircraft to be refueled, is received from a single camera. An initial position estimate is determined for the aircraft for the plurality of video frames, generating an estimated flight history for the aircraft. The estimated flight history for the aircraft is used to determine a temporally consistent refined position estimate, based on known aircraft flight path trajectories in an aerial refueling setting. The position of a fuel receptacle on the aircraft is determined, based on the refined position estimate for the aircraft, and an aerial refueling boom may be controlled to engage the fuel receptacle. Examples may use a deep learning neural network (NN) or optimization (e.g., bundle adjustment) to determine the refined position estimate from the estimated flight history.

First claim

What is claimed is:

1. A method of aerial refueling, the method comprising:
receiving a video stream showing an aircraft to be refueled, wherein the video stream comprises a plurality of video frames, each video frame showing the aircraft;
determining, for each video frame of the plurality of video frames, an initial position estimate for the aircraft, wherein the initial position estimates for the plurality of video frames comprises an estimated flight history for the aircraft;
based on at least the estimated flight history for the aircraft, determining, by an estimate refiner, a temporally consistent refined position estimate for the aircraft;
based on at least the refined position estimate for the aircraft, determining a position of a fuel receptacle on the aircraft;
determining a position of a boom tip of an aerial refueling boom, wherein the aerial refueling boom is telescopically extendable;
tracking a distance between the boom tip and the fuel receptacle; and
based on at least the position of the fuel receptacle and the position of the boom tip, controlling the aerial refueling boom, using boom control parameters, to engage the fuel receptacle, wherein the boom control parameters include a parameter for telescopically extending the aerial refueling boom.

2. The method of claim 1 , wherein the video stream is provided by a single camera.

3. The method of claim 1 , further comprising:
based on at least the estimated flight history for the aircraft, determining, by the estimate refiner, position estimation refinement parameters, wherein the refinement parameters comprise a translation refinement and a rotational refinement, and wherein determining the temporally consistent refined position estimate comprises determining the temporally consistent refined position estimate based on at least the estimated flight history and the position estimation refinement parameters.

4. The method of claim 1 , wherein the estimate refiner comprises a first neural network (NN).

5. The method of claim 1 , wherein the estimate refiner comprises an optimizer.

6. The method of claim 1 , further comprising:
generating an overlay image comprising:
a model projection, based on at least an aircraft model and the refined position estimate; and
the video frame showing the aircraft or a subsequent video frame showing the aircraft; and

displaying the overlay image.

7. A system for aerial refueling, the system comprising:
one or more processors; and
a memory storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising:
receiving a video stream showing an aircraft to be refueled, wherein the video stream comprises a plurality of video frames, each video frame showing the aircraft;
determining, for each video frame of the plurality of video frames, an initial position estimate for the aircraft, wherein the initial position estimates for the plurality of video frames comprises an estimated flight history for the aircraft;
based on at least the estimated flight history for the aircraft, determining, by an estimate refiner, a temporally consistent refined position estimate for the aircraft;
based on at least the refined position estimate for the aircraft, determining a position of a fuel receptacle on the aircraft;
determining a position of a boom tip of an aerial refueling boom, wherein the aerial refueling boom is telescopically extendable;
tracking a distance between the boom tip and the fuel receptacle; and
based on at least the position of the fuel receptacle and the position of the boom tip, controlling the aerial refueling boom, using boom control parameters, to engage the fuel receptacle, wherein the boom control parameters include a parameter for telescopically extending the aerial refueling boom.

8. The system of claim 7 , wherein the video frame is provided by a single camera.

9. The system of claim 7 , wherein the operations further comprise:
based on at least the estimated flight history for the aircraft, determining, by the estimate refiner, position estimation refinement parameters, wherein the refinement parameters comprise a translation refinement and a rotational refinement, and wherein determining the temporally consistent refined position estimate comprises determining the temporally consistent refined position estimate based on at least the estimated flight history and the position estimation refinement parameters.

10. The system of claim 7 , wherein the estimate refiner comprises a first neural network (NN).

11. The system of claim 7 , wherein the estimate refiner comprises an optimizer.

12. The system of claim 7 , further comprising:
a boom control, wherein the processor is to perform the operation of controlling the aerial refueling boom to engage the fuel receptacle via the boom control.

13. A computer program product, comprising a non-transitory computer usable medium having a computer readable program code embodied therein, the computer readable program code adapted to be executed to implement a method of aerial refueling, the method comprising:
receiving a video stream showing an aircraft to be refueled, wherein the video stream comprises a plurality of video frames, each video frame showing the aircraft;
determining, for each video frame of the plurality of video frames, an initial position estimate for the aircraft, wherein the initial position estimates for the plurality of video frames comprises an estimated flight history for the aircraft;
based on at least the estimated flight history for the aircraft, determining, by an estimate refiner, a temporally consistent refined position estimate for the aircraft;
based on at least the refined position estimate for the aircraft, determining a position of a fuel receptacle on the aircraft;
determining a position of a boom tip of an aerial refueling boom, wherein the aerial refueling boom is telescopically extendable;
tracking a distance between the boom tip and the fuel receptacle; and
based on at least the position of the fuel receptacle and the position of the boom tip, controlling the aerial refueling boom, using boom control parameters, to engage the fuel receptacle, wherein the boom control parameters include a parameter for telescopically extending the aerial refueling boom.

14. The computer program product of claim 13 , wherein the video frame is provided by a single camera.

15. The computer program product of claim 13 , wherein the method further comprises:
based on at least the estimated flight history for the aircraft, determining, by the estimate refiner, position estimation refinement parameters, wherein the refinement parameters comprise a translation refinement and a rotational refinement, and wherein determining the temporally consistent refined position estimate comprises determining the temporally consistent refined position estimate based on at least the estimated flight history and the position estimation refinement parameters.

16. The computer program product of claim 13 , wherein the estimate refiner comprises a first neural network (NN).

17. The computer program product of claim 13 , wherein the estimate refiner comprises an optimizer.

18. The method of claim 1 , wherein the estimate refiner produces the refined position estimate by comparing the estimated flight history with known aircraft flight path trajectories for refueling aircraft as viewed by a refueling platform.

19. The system of claim 7 , wherein the estimate refiner produces the refined positio

Assignees

• Boeing Co

Inventors

• Nguyen Leon Nhat
• Smith Haden Harrison
• Hung Fan Hin
• Khosla Deepak

Classifications

• G05D1/695

  for maintaining a fixed relative position of the vehicles, e.g. for convoy travelling or formation flight · CPC title

• G06T19/006

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• B64D39/00 Primary

  Refuelling during flight · CPC title

• G05D1/104

  involving a plurality of aircrafts, e.g. formation flying (traffic control systems for aircraft G08G5/00) · CPC title

• G03B15/006

  Apparatus mounted on flying objects · CPC title