System and method for automatic registration of 3D data with electro-optical imagery via photogrammetric bundle adjustment

Patent metadata

Field	Value
Publication number	US-9275267-B2
Application number	US-201314053990-A
Country	US
Kind code	B2
Filing date	Oct 15, 2013
Priority date	Oct 23, 2012
Publication date	Mar 1, 2016
Grant date	Mar 1, 2016

Abstract

Accurate automatic registration and fusion of LADAR (from laser detection and ranging) and EO (electro-optical) data from different sensors provides additional analysis and exploitation value beyond what each data set provides on its own. Such data sets often exhibit significant misregistration due to uncorrelated geometric errors between or among two or more sensors. One or more automatic algorithms achieve superior registration as well as algorithms for fusing the data in three dimensions (3D). The fused data can provide multi-image colorization for change detection, automatic generation of surface relief colorization, interactive and/or automatic filtering of 3D vegetation points for LADAR foliage penetration analysis, automatic surface orientation determination for improved spectroradiometric exploitation, and other benefits that cannot be achieved by the LADAR or EO data alone.

First claim

What is claimed is:

1. A system for automatically registering 3D data with a multispectral image, the system comprising processing circuitry arranged to:
obtain a panchromatic-sharpened multispectral image having image coordinates;
extract ground control point coordinates (GCPs) from the image and from a set of corresponding 3D data, the 3D data including a plurality of 3D geodetic coordinates;
input the ground control points into a photogrammetric bundle adjustment (BA);
generate at least one set of corrected rational polynomial coefficients from the photogrammetric bundle adjustment; and
colorize the 3D data with the corrected rational polynomial coefficients to generate and store red, green, and blue values for each 3D geodetic coordinate.

2. The system of claim 1 , wherein the processing circuitry is further arranged to display the red, green, and blue values as colorized values for the 3D data.

3. The system of claim 1 , wherein colorizing the 3D geodetic coordinates comprises:
converting the image coordinates to corresponding 3D geodetic coordinates for occlusion determination;
evaluating the corresponding 3D geodetic coordinates via the corrected rational polynomial coefficients;
interpolating image intensities surrounding sample and line coordinates to obtain the red, green, and blue values for each 3D geodetic coordinate; and
storing the red, green, and blue values for each 3D geodetic coordinate.

4. The system of claim 3 , wherein the red, green, and blue values are stored as extra fields for each 3D geodetic coordinate.

5. The system of claim 3 , wherein the processing circuitry is further arranged to:
for each image coordinate, calculate a Normalized Difference Vegetative Index (NDVI) as (NIR minus RED) divided by (NIR plus RED), where NIR is an intensity value from a near-infrared image in the multispectral image, and RED is an intensity value from a red image in the multispectral image;
projecting the NDVI values to PAN space to form NDVI data;
colorizing the NDVI data with the corrected rational polynomial coefficients to generate and store NDVI values for each 3D geodetic coordinate; and
displaying the NDVI values for the 3D data.

6. The system of claim 1 , wherein the panchromatic-sharpened multispectral image is produced by a satellite.

7. The system of claim 1 , wherein the at least one set of 3D geodetic coordinates is obtained from laser detection and ranging (LADAR).

8. The system of claim 1 , wherein extracting ground control points (GCPs) from the image coordinates and from the set of corresponding 3D data comprises:
rasterizing the LADAR intensity data into an image;
automatically finding features in the LADAR intensity data that are conducive to correlation;
performing automatic correlations at the feature locations with the EO image intensities;
producing a correspondence of LADAR raster coordinates to EO image coordinates;
converting the LADAR raster coordinates to X,Y coordinates;
evaluating the 3D data to obtain a corresponding Z coordinate;
converting UTM coordinates to geodetic coordinates, which correspond to EO image coordinates (s 2 , l 2 );
using the automatically-determined ground coordinates as GCPs in a photogrammetric bundle adjustment of the EO image(s).

9. The system of claim 1 , wherein extracting ground control points (GCPs) from the image coordinates and from the set of corresponding 3D data comprises:
assigning to each 3D data point an intensity that is proportional to the dot product of a sun vector angle and a surface local unit normal vector at the 3D data point;
modifying the intensity values via a simple atmospheric scattering model;
performing a correlation process at a grid of points between projected SSR and EO images; and
retaining good correlation points via correlation quality thresholding.

10. A method for automatically registering 3D data with a multispectral image, the method comprising:
obtaining a panchromatic-sharpened multispectral image having image coordinates;
extracting ground control points (GCPs) from the image coordinates and from a set of corresponding 3D data, the 3D data including a plurality of 3D geodetic coordinates;
inputting the ground control points into a photogrammetric bundle adjustment (BA);
generating at least one set of corrected rational polynomial coefficients from the photogrammetric bundle adjustment; and
colorizing the 3D data with the corrected rational polynomial coefficients to generate and store red, green, and blue values for each 3D geodetic coordinate.

11. The method of claim 10 , further comprising displaying the red, green, and blue values as colorized values for the 3D data.

12. The method of claim 10 , wherein colorizing the 3D geodetic coordinates comprises:
converting the image coordinates to corresponding 3D geodetic coordinates;
evaluating the corresponding 3D geodetic coordinates via the corrected rational polynomial coefficients;
interpolating image intensities surrounding sample and line coordinates to obtain the red, green, and blue values for each 3D geodetic coordinate; and
storing the red, green, and blue values for each 3D geodetic coordinate.

13. The method of claim 12 , wherein the red, green, and blue values are stored as extra fields for each 3D geodetic coordinate.

14. The method of claim 12 , further comprising:
for each image coordinate, calculate a Normalized Difference Vegetative Index (NDVI) as (NIR minus RED) divided by (NIR plus RED), where NIR is an intensity value from a near-infrared image in the multispectral image, and RED is an intensity value from a red image in the multispectral image;
projecting the NDVI values to PAN space to form NDVI data;
colorizing the NDVI data with the corrected rational polynomial coefficients to generate and store NDVI values for each 3D geodetic coordinate; and
displaying the NDVI values for the 3D data.

15. The method of claim 12 , wherein the panchromatic-sharpened multispectral image is produced by a satellite.

16. The method of claim 10 , wherein the at least one set of 3D geodetic coordinates is obtained from laser detection and ranging (LADAR).

17. The method of claim 10 , wherein extracting ground control points (GCPs) from the image coordinates and from the set of corresponding 3D data comprises:
rasterizing the LADAR intensity data into an image;
automatically finding features in the LADAR intensity data that are conducive to correlation;
performing automatic correlations at the feature locations with the EO image intensities;
producing a correspondence of LADAR raster coordinates to EO image coordinates;
converting the LADAR raster coordinates to X,Y coordinates;
evaluating the 3D data to obtain a corresponding Z coordinate;
converting UTM coordinates to geodetic coordinates, which correspond to EO image coordinates (s 2 , l 2 );
using the automatically-determined ground coordinates as GCPs in a photogrammetric bundle adjustment of the EO image(s).

18. The method of claim 10 , wherein extracting ground control points (GCPs) from the image coordinates and from the set of corresponding 3D data comprises:
assigning to each 3D data point an intensity that is proportional to the dot product of a sun vector angle and a surface local unit normal vector at the 3D data point;
modifying the intensity values via a simple atmospheric scattering model;
performing a correlation process at a grid of points between projected SSR and EO images; and
retaining good correlation points via correlation quality thresholding.

Assignees

• Raytheon Co

Inventors

• Verret Jody D

Classifications

• G06V20/13

  Satellite images · CPC title

• G06K9/0063

  Physics · mapped topic

• G06K9/00201 Primary

  Physics · mapped topic

• G06V20/64 Primary

  Three-dimensional [3D] objects · CPC title