Real Time Multi Dimensional Image Fusing

Patent metadata

Field	Value
Publication number	US-2016266256-A1
Application number	US-201514644315-A
Country	US
Kind code	A1
Filing date	Mar 11, 2015
Priority date	Mar 11, 2015
Publication date	Sep 15, 2016
Grant date	—

Abstract

Within examples, systems and methods of generating a synthetic image representative of an environment of a vehicle are described comprising generating a first image using infrared information from an infrared (IR) camera, generating a second image using laser point cloud data from a LIDAR, generating an embedded point cloud representative of the environment based on a combination of the first image and the second image, receiving navigation information traversed by the vehicle, transforming the embedded point cloud into a geo-referenced coordinate space based on the navigation information, and combining the transformed embedded point cloud with imagery of terrain of the environment to generate the synthetic image representative of the environment of the vehicle.

First claim

What is claimed is:

1 . A method of generating a synthetic image representative of an environment of a vehicle, the method comprising:
generating a first image of the environment using infrared information from an infrared (IR) camera on the vehicle; generating a second image of the environment using laser point cloud data from a LIDAR on the vehicle; generating an embedded point cloud representative of the environment based on a combination of the first image and the second image such that additional data is embedded into the laser point cloud data; receiving navigation information of the environment traversed by the vehicle from data stored in a navigation database; transforming the embedded point cloud into a geo-referenced coordinate space based on the navigation information; and combining the transformed embedded point cloud with imagery of terrain of the environment to generate the synthetic image representative of the environment of the vehicle.

2 . The method of claim 1 , wherein the first image, the second image, and the navigation information are each generated and received at approximately the same point in time.

3 . The method of claim 1 , wherein generating the embedded point cloud representative of the environment based on the combination of the first image and the second image comprises:
for each laser data point of the laser point cloud data from the LIDAR, projecting the laser data point into a corresponding pixel location of the first image so as to map the laser point cloud data onto the first image.

4 . The method of claim 1 , wherein transforming the embedded point cloud data into the geo-referenced coordinate space based on the navigation information comprises:
adjusting the embedded point cloud data according to the IR camera and LIDAR attitude and linear offsets relative a navigation system of the vehicle, wherein adjustments are made using six degree of freedom (DOF) platform data including latitude, longitude, altitude, heading, pitch, and roll from the navigation system of the vehicle.

5 . The method of claim 1 , wherein combining the transformed embedded point cloud with imagery of terrain of the environment comprises:
overlaying an image of the terrain onto the transformed embedded point cloud.

6 . The method of claim 1 , further comprising displaying the synthetic image of the terrain of the environment on a multi-function display (MFD) of the vehicle.

7 . The method of claim 1 , further comprising displaying the synthetic image of the terrain of the environment on a head mounted display (HMD).

8 . The method of claim 1 , further comprising performing the method in real-time as the infrared information is received from the IR camera and the laser point cloud data is received from the LIDAR during operation of the vehicle traversing the environment.

9 . The method of claim 1 , further comprising, while the vehicle is traversing the environment:
receiving the infrared information from the IR camera; receiving the laser point cloud data from the LIDAR; receiving the navigation information from a navigation system on the vehicle; performing the method in real-time to generate the synthetic image representative of the environment of the vehicle while the vehicle is traversing the environment; and displaying the synthetic image of the terrain of the environment on a display of the vehicle.

10 . The method of claim 1 , further comprising:
generating the synthetic image based on the vehicle operating in a degraded visual environment (DVE) including near-zero to zero visibility conditions.

11 . The method of claim 1 , further comprising:
determining a level of obscuration of the embedded point cloud; and adjusting parameters of one or more of the IR camera and the LIDAR on the vehicle based on the level of obscuration for adaptive feedback control of sensor parameters.

12 . The method of claim 1 , further comprising:
storing the infrared information from the IR camera and the laser point cloud data from the LIDAR into a dataset; determining, as data of the infrared information from the IR camera and the laser point cloud data from the LIDAR is received, whether the data indicates additional spatial resolution of a representation of the environment; based on the data indicating the additional spatial resolution, generating a new higher resolution data point in the dataset; and based on the data not indicating the additional spatial resolution, evaluating the data for update.

13 . The method of claim 12 , wherein determining whether the data indicates additional spatial resolution of the representation of the environment comprises:
determining whether the data is of an alternate range or viewpoint to a targeted object or area.

14 . The method of claim 12 , wherein determining whether the data indicates additional spatial resolution of the representation of the environment comprises:
determining sensor measurement accuracy, wherein the sensor measurement accuracy is based on a level of obscuration of the environment.

15 . A non-transitory computer readable medium having stored thereon instructions that, upon executed by a computing device, cause the computing device to perform functions comprising:
generating a first image of the environment using infrared information collected from an infrared (IR) camera on an vehicle; generating a second image of the environment using laser point cloud data collected from a LIDAR on the vehicle; generating an embedded point cloud representative of the environment based on a combination of the first image and the second image such that additional data is embedded into the laser point cloud data; receiving navigation information of the environment traversed by the vehicle from data stored in a navigation database; transforming the embedded point cloud into a geo-referenced coordinate space based on the navigation information; and combining the transformed embedded point cloud with imagery of terrain of the environment to generate the synthetic image representative of the environment of the vehicle.

16 . The non-transitory computer readable medium of claim 15 , wherein combining the transformed embedded point cloud with imagery of terrain of the environment comprises:
overlaying an image of the terrain onto the transformed embedded point cloud.

17 . The non-transitory computer readable medium of claim 15 , further comprising performing the functions in real-time as the infrared information is received from the IR camera and the laser point cloud data is received from the LIDAR during operation of the vehicle traversing the environment.

18 . A system comprising:
an infrared (IR) camera to collect infrared information of an environment of an vehicle; a LIDAR to collect laser point cloud data of the environment of the vehicle; a navigation system configured to determine navigation information of the vehicle; a processor to generate a synthetic representation of the environment of the vehicle, in real-time while the vehicle is traversing the environment, based on outputs of the IR camera, the LIDAR, and the navigation system, wherein generation of the synthetic representation comprises:
generating an embedded point cloud representative of the environment based on a combination of the infrared information and the laser point cloud data such that additional data is embedded into the laser point cloud data; and
combining the embedded point cloud with imagery of terrain of the environment; and
a display to display the synth

Assignees

• Boeing Co

Inventors

• Allen Robert C
• Blanton Brendan
• Gresko Lauren
• Mitra Rajib
• Vincent Frank

Classifications

• G06V20/56

  exterior to a vehicle by using sensors mounted on the vehicle · CPC title

• G01S17/89 Primary

  for mapping or imaging · CPC title

• G08G5/74

  for monitoring terrain · CPC title

• H04N5/33

  Transforming infrared radiation (cameras or camera modules for generating image signals from infrared radiation H04N23/20; circuitry of SSIS for transforming infrared radiation into image signals H04N25/20) · CPC title

• G01S17/87

  Combinations of systems using electromagnetic waves other than radio waves · CPC title