Stereo camera system for collision avoidance during aircraft surface operations

Patent metadata

Field	Value
Publication number	US-11094208-B2
Application number	US-201615281189-A
Country	US
Kind code	B2
Filing date	Sep 30, 2016
Priority date	Sep 30, 2016
Publication date	Aug 17, 2021
Grant date	Aug 17, 2021

Abstract

A collision avoidance system comprises a pair of video cameras mounted to a vertical stabilizer of the aircraft, a machine vision processing unit, and a system to inform the pilots of a potential collision. The machine vision processing unit is configured to process image data captured by the video cameras using stereoscopic and structure from motion techniques to detect an obstacle that is near or in the path of the aircraft. Estimates of the range to the object and the rate of change of that range are computed. With the range and range rate, a time to collision can be estimated toward every point of the aircraft. A pilot warning can be sounded based on the nearness of the potential collision. A method of calibrating the video cameras using existing feature points on the top of the aircraft is initiated in response to power being turned on.

First claim

The invention claimed is:

1. A system for warning a pilot of a risk of collision, the system comprising:
a first camera mounted at a first height to a leading edge of a vertical stabilizer of an aircraft for generating a first stream of video frames that include first image data representing an image of an object in a scene and second image data representing an image of a portion of the aircraft;
a second camera mounted at a second height to the leading edge of the vertical stabilizer for generating a second stream of video frames that include third image data representing an image of the object in the scene and fourth image data representing an image of the portion of the aircraft, wherein the second height is less than the first height;
a cue system on a flight deck of the aircraft capable of generating a cue; and
a computer system programmed to:
process the video frames of the first and second streams to determine a first range to the object and a first range rate at which the range to the object is changing at a first time;
compute a first time to collision based on at least the first range and first range rate; and
trigger the cue system to produce a first cue in response to the first time to collision being less than a first detection threshold.

2. The system as recited in claim 1 , wherein the computer system is further programmed to:
process the video frames of the first and second streams to determine a second range to the object and a second range rate at which the range is changing at a second time that is subsequent to the first time;
compute a second time to collision based on at least the second range and second range rate; and
trigger the cue system to produce a second cue different than the first cue in response to the second time to collision being less than a second detection threshold, wherein the second detection threshold is less than the first detection threshold.

3. The system as recited in claim 2 , wherein the first cue is a sound having a first volume and the second cue is a sound having a second volume greater than the first volume.

4. The system as recited in claim 2 , wherein the first cue is a repetitive sound having a first repetition rate and the second cue is a repetitive sound having a second repetition rate greater than the first repetition rate.

5. The system as recited in claim 1 , wherein the video frames of the first and second streams are processed using a stereoscopic technique to estimate a depth of the object.

6. The system as recited in claim 1 , wherein the video frames of the first and second streams are processed using a structure from motion technique to estimate a three-dimensional structure of the scene.

7. The system as recited in claim 1 , wherein the first and second cameras are directed in a forward direction along a centerline of the aircraft with respective fields of view that are at least partially overlapping.

8. The system as recited in claim 1 , wherein the computer system is further programmed to:
activate the first and second cameras to capture first and second images; and
calibrate the first and second cameras based on the first and second images.

9. The system as recited in claim 8 , wherein the second camera has markings which are in a field of view of the first camera and the computer system is further programmed to:
detect image data correlated to a camera template in the first image, said camera template including image data representing the markings on the second camera; and
declare the first camera to be a top camera in response to detection of image data correlated to the camera template in the first image.

10. The system as recited in claim 1 , wherein the computer system is further programmed to:
segment the image data in the first and second images which represents portions of the aircraft appearing in both images using feature locations with sizes appropriate for image templates of each feature;
correlate each feature segment against its template image;
compute a maximum correlation coefficient together with an offset from an original feature location;
compare the maximum correlation coefficient to a correlation coefficient threshold; and
determine an essential matrix based at least in part on the results of the comparison of the maximum correlation coefficient to the correlation coefficient threshold.

11. A system for warning a pilot of a risk of collision, the system comprising:
a first camera mounted at a first height to a leading edge of a vertical stabilizer of an aircraft;
a second camera mounted at a second height to the leading edge of the vertical stabilizer, wherein the second height is lower than the first height, said second camera having markings which are in a field of view of the first camera; and
a computer system programmed to:
activate the first and second cameras to capture first and second images respectively;
detect image data correlated to a camera template in the first image, said camera template including image data representing the markings on the second camera; and
declare the first camera to be a top camera in response to detection of the camera template in the first image.

12. A method for avoiding a collision during ground maneuvering by an aircraft using cameras and an onboard computer system operably coupled to the cameras and configured to capture and process image frames from the cameras, the method comprising:
maneuvering an aircraft on the ground;
activating a first camera mounted at a first height to a leading edge of a vertical stabilizer of the aircraft to generate a first stream of video frames during the maneuvering that include first image data representing an image of an object in a scene and second image data representing an image of a portion of the aircraft;
activating a second camera mounted at a second height to the leading edge of the vertical stabilizer to generate a second stream of video frames during the maneuvering that include third image data representing an image of the object in the scene and fourth image data representing an image of the portion of the aircraft, wherein the second height is less than the first height;
using the computer system to process the video frames of the first and second streams to determine a first range to the object and a first range rate at which the range to the object is changing at a first time;
using the computer system to compute a first time to collision based on at least the first range and first range rate;
producing a first cue in response to the first time to collision being less than a first detection threshold; and
activating brakes on the aircraft in response to production of the first cue.

13. The method as recited in claim 12 , further comprising:
using the computer system to process the video frames of the first and second streams to determine a second range to the object and a second range rate at which the range is changing at a second time that is subsequent to the first time;
using the computer system to compute a second time to collision based on at least the second range and second range rate; and
producing a second cue different than the first cue in response to the second time to collision being less than a second detection threshold, wherein the second detection threshold is less than the first detection threshold.

14. The method as recited in claim 13 , wherein the first cue is a sound having a first volume and the second cue is a sound having a second volume greater than the first volume.

15. The method as recited in claim 13 , wherein the first cue is a repetitive sound having a first r

Assignees

• Boeing Co

Inventors

• Ray Gary Alan
• Gilbert Brian David

Classifications

• H04N13/239 Primary

  using two two-dimensional [2D] image sensors having a relative position equal to or related to the interocular distance (H04N13/243 takes precedence) · CPC title

• G08G5/80

  Anti-collision systems · CPC title

• G08G5/51

  for control when on the ground, e.g. taxiing or rolling · CPC title

• G08G5/21

  located onboard the aircraft · CPC title

• B64D45/00 Primary

  Aircraft indicators or protectors not otherwise provided for · CPC title