Multi-source image fusion

Patent metadata

Field	Value
Publication number	US-10970815-B2
Application number	US-201916364969-A
Country	US
Kind code	B2
Filing date	Mar 26, 2019
Priority date	Jul 10, 2018
Publication date	Apr 6, 2021
Grant date	Apr 6, 2021

Abstract

A method can include registering a first image of a region to a three-dimensional (3D) point set of the region to generate a registered first image, registering a second image of the region to the 3D point set to generate a registered second image, identifying, based on the 3D point set, geometric tie points of the registered first image and the registered second image, projecting, using an affine transformation determined based on the identified geometric tie points, pixels of the registered first image to an image space of the registered second image to generate a registered and transformed first image, and displaying the registered and transformed first image and the registered second image simultaneously.

First claim

What is claimed is:

1. A method for fusing images, the method comprising:
adjusting, using ground control points (GCPs), image geometry in metadata of a two-dimensional (2D) first image of a region to match image geometry of a three-dimensional (3D) point set of at least a portion of the region to generate a registered first image, the GCS independent of the 3D point set;
adjusting, using the GCPs, image geometry in metadata of a 2D second image of at least the portion of the region to match the image geometry of the 3D point set to generate a registered second image;
identifying, based on the 3D point set, geometric tie points of the registered first image and the registered second image; and
projecting, using an affine transformation determined based on a least squares fit of the identified geometric tie points, pixels of the registered first image to an image space of the registered second image to generate a registered and transformed first image.

2. The method of claim 1 , further comprising displaying the registered and transformed first image and the registered second image simultaneously.

3. The method of claim 1 , wherein the 2D first image is generated using a first optical device and the 2D second image is generated using a second, different optical device.

4. The method of claim 1 , wherein the 2D first image is of a first image type including one of an electro-optical (EO) image, a synthetic aperture radar (SAR) image, an infrared (IR) image, and a multi-spectral image, and the 2D second image is of a second, different image type.

5. The method of claim 4 , wherein the geometric tie points are situated in a regular grid on the registered first and second images.

6. The method of claim 5 , wherein x, y points of the grid are projected to the image space of the registered first image and the registered second image to generate geometric tie point coordinates between the images.

7. The method of claim 6 , further comprising generating the affine transformation to project the geometric tie point coordinates in the registered first image to the geometric tie point coordinates in the registered second image.

8. The method of claim 1 , wherein the geometric tie points are at substantially a same elevation based on the 3D point set.

9. The method of claim 1 , further comprising determining a corresponding pixel in the registered image to which a transformed projected first image pixel of the transformed projected first image pixels corresponds using a bilinear interpolation.

10. The method of claim 1 , wherein the 2D first image and the 2D second image are captured from different perspectives.

11. A system for fusing images, the system comprising:
processing circuitry configured to:
adjust, using ground control points (GCPs), image geometry in metadata of a two-dimensional (2D) first image of a region to match image geometry of a three-dimensional (3D) point set of the region to generate a registered first image, the GCPs independent of the 3D point set;
adjust, using the GCPs, image geometry in metadata of a 2D second image of the region to match image geometry of the 3D point set to generate a registered second image;
identifying, based on the 3D point set, geometric tie points of the registered first image and the registered second image;
projecting, using an affine transformation determined based on a least squares fit of the identified geometric tie points, pixels of the registered first image to an image space of the registered second image to generate a registered and transformed first image; and
a display to provide a view of the registered and transformed first image and the registered second image simultaneously.

12. The system of claim 11 , wherein the 2D first image is generated using a first optical device and the 2D second image is generated using a second, different optical device.

13. The system of claim 11 , wherein the 2D first image is of a first image type including one of an electro-optical (EO) image, a synthetic aperture radar (SAR) image, an infrared (IR) image, and a multi-spectral image, and the 2D second image is of a second, different image type.

14. The system of claim 11 , wherein the geometric tie points are at a same elevation determined based on the 3D point set.

15. The system of claim 14 , wherein the geometric tie points are situated in a regular grid on the registered first and second images.

16. The system of claim 15 , wherein x, y points of the grid are projected to the image space of the registered first image and the registered second image to generate geometric tie point coordinates between the images.

17. At least one non-transitory machine-readable medium including instructions, that when executed by a machine, configure the machine to perform operations comprising:
adjusting, using ground control points (GCPs), image geometry in metadata of a two-dimensional (2D) first image of a region to match image geometry of a three-dimensional (3D) point set of the region to generate a registered first image, the GCPs independent from the 3D point set;
adjusting, using the GCPs, image geometry in metadata of a 2D second image of the region to match image geometry of the 3D point set to generate a registered second image;
identifying, based on the 3D point set, geometric tie points of the registered first image and the registered second image;
projecting, using an affine transformation determined based on a least squares fit of the identified geometric tie points, pixels of the registered first image to an image space of the registered second image to generate a registered and transformed first image; and
providing signals that cause a display to provide a view of the registered and transformed first image and the registered second image simultaneously.

18. The at least one non-transitory machine-readable medium of claim 17 , wherein the geometric tie points are at substantially a same elevation based on the 3D point set and situated in a regular grid on the registered first and second images.

19. The at least one non-transitory machine-readable medium of claim 18 , wherein x,y points of the grid are projected to the image space of the registered first image and the registered second image to generate geometric tie point coordinates between the images, and the operations further include generating the affine transformation to project the geometric tie point coordinates in the registered first image to the geometric tie point coordinates in the registered second image.

20. The at least one non-transitory machine-readable medium of claim 17 , wherein the operations further comprise determining a corresponding pixel in the registered first image to which a transformed projected first image pixel of the transformed projected first image pixels corresponds using a bilinear interpolation.

Assignees

• Raytheon Co

Inventors

• Ely Richard W

Classifications

• G06T2207/10048

  Infrared image · CPC title

• G06T2207/30181

  Earth observation · CPC title

• G06T3/4038

  Image mosaicing, e.g. composing plane images from plane sub-images · CPC title

• G06T2207/10032

  Satellite or aerial image; Remote sensing · CPC title

• G06T7/30

  Determination of transform parameters for the alignment of images, i.e. image registration · CPC title