Automatic co-registration of thermal and visible image pairs

Patent metadata

Field	Value
Publication number	US-11869204-B2
Application number	US-202117306891-A
Country	US
Kind code	B2
Filing date	May 3, 2021
Priority date	Nov 6, 2018
Publication date	Jan 9, 2024
Grant date	Jan 9, 2024

Abstract

Image pair co-registration systems and methods include receiving a pair of multi-modal images, defining a parametric deformation model, defining a loss function that is minimized when the pair of images are aligned, and performing a multi-scale search to determine deformation parameters that minimize the loss function. The optimized deformation parameters define an alignment of the pair of images. The pair of images may include visible spectrum image and an infrared image. The method further includes resizing the visible spectrum image to match the infrared image, applying at least one lens distortion correction model, and normalizing a dynamic range of each of the pair of images. The multi-scale search may further include resizing the pair of images to a current processing scale, applying adaptive histogram equalization to the pair of images to generate equalized images, applying Gaussian Blur to the equalized images, and optimizing the deformation parameters.

First claim

What is claimed is:

1. A method comprising:
receiving a pair of images;
defining a parametric deformation model providing an image deformation for each of a set of deformation parameters;
defining a loss function based, at least in part, on at least one directional gradient of each of a processed pair of images obtained from the received pair of images by a process comprising applying the parametric deformation model, wherein the loss function is minimized when the pair of images are aligned; and
performing a multi-scale search to determine an optimal deformation parameter that minimizes the loss function;
wherein the optimal deformation parameter defines an alignment of the pair of images.

2. The method of claim 1 , wherein the pair of images comprises a visible spectrum image and an infrared image.

3. The method of claim 2 , wherein the process further comprises resizing the visible spectrum image to match the infrared image;
wherein the loss function is based, at least in part, on comparing the directional gradients of the processed pair of images at each of a plurality of pixels.

4. The method of claim 1 , wherein the process further comprises applying at least one lens distortion correction model to each of the pair of images.

5. The method of claim 1 , wherein the process further comprises normalizing a dynamic range of each of the pair of images; and
wherein the loss function is based, at least in part, on comparing the directional gradients of the processed pair of images in each of a plurality of directions at each of the plurality of pixels.

6. The method of claim 1 , wherein the process further comprises resizing the pair of images to a current processing scale of the multi-scale search.

7. The method of claim 6 , wherein the process further comprises, after resizing the pair of images to a current processing scale of the multi-scale search, applying adaptive histogram equalization to the pair of images to generate equalized images.

8. The method of claim 7 , wherein the process further comprises applying Gaussian Blur to the equalized images.

9. The method of claim 1 , wherein the loss function is based, at least in part, on determining a sum of squares of differences between absolute values of the directional gradients of the processed pair of images in each of a plurality of directions at each of a plurality of pixels.

10. The method of claim 1 , wherein:
each deformation parameter p is defined by p=[δx,δy,s,ϕ], where s is a scaling factor that controls image scaling, ϕ controls image rotation in degrees, and δx and δy control the image translation (in pixels) in horizontal and vertical directions, respectively; and
for each deformation parameter, the loss function is based, at least in part, on a penalty term on the deformation parameter.

11. A system comprising:
a processor; and
a local memory operable to store a plurality of machine readable instructions which when executed by the processor are operable to cause the system to perform steps comprising:
receiving a pair of images;
defining a parametric deformation model providing an image deformation for each of a set of deformation parameters;
defining a loss function based, at least in part, on at least one directional gradient of each of a processed pair of images obtained from the received pair of images by a process comprising applying the parametric deformation model, wherein the loss function is minimized when the pair of images are aligned; and
performing a multi-scale search to determine an optimal deformation parameter that minimizes the loss function;
wherein the optimal deformation parameter defines an alignment of the pair of images.

12. The system of claim 11 , wherein the pair of images comprises a visible spectrum image and an infrared image; and
wherein the system further comprises a plurality of image capture components, including a visible spectrum sensor operable to capture the visible spectrum image, and an infrared sensor operable to capture the infrared image.

13. The system of claim 12 , further comprising optical components arranged to pass an image of a scene to the image capture components; and
wherein the process further comprises applying at least one lens distortion correction model to each of the pair of images, the lens distortion correction model comprising a model of the optical components.

14. The system of claim 11 , wherein:
the loss function is based, at least in part, on comparing the directional gradients of the processed pair of images in each of a plurality of directions; and
the process further comprises:
normalizing a dynamic range of each of the pair of images.

15. The system of claim 11 , wherein:
the loss function is based, at least in part, on comparing the directional gradients of the processed pair of images at each of a plurality of pixels; and
the process further comprises:
resizing the pair of images to a current processing scale of the multi-scale search.

16. The system of claim 15 , wherein the process further comprises, after resizing the pair of images to a corresponding processing scale of the multi-scale search, applying adaptive histogram equalization to the pair of images to generate equalized images.

17. The system of claim 16 , wherein the process further comprises applying Gaussian Blur to the equalized images.

18. The system of claim 11 , wherein the loss function is based, at least in part, on determining a sum of squares of differences between absolute values of the directional gradients of the processed pair of images in each of a plurality of directions at each of a plurality of pixels.

19. The system of claim 11 , wherein each deformation parameter p is defined by p=[δx,δy,s,ϕ], where s is a scaling factor that controls image scaling, ϕ controls image rotation in degrees, and δx and δy control the image translation (in pixels) in horizontal and vertical directions, respectively; and
for each deformation parameter, the loss function is based, at least in part, on a penalty term on the deformation parameter.

20. An unmanned aerial system comprising the system of claim 11 .

Assignees

• Teledyne Flir Commercial Systems Inc

Inventors

• Sanchez-Monge Enrique
• Foi Alessandro
• Kent Michael
• Messely Pieter

Classifications

• G06T7/337 Primary

  involving reference images or patches · CPC title

• B64C39/024

  of the remote controlled vehicle type, i.e. RPV · CPC title

• G06T3/40

  Scaling of whole images or parts thereof, e.g. expanding or contracting · CPC title

• G06T5/006

  Physics · mapped topic

• G06T5/007

  Physics · mapped topic