Systems and methods for dynamic registration of multimodal images

Patent metadata

Field	Value
Publication number	US-9684963-B2
Application number	US-201514986571-A
Country	US
Kind code	B2
Filing date	Dec 31, 2015
Priority date	Dec 31, 2014
Publication date	Jun 20, 2017
Grant date	Jun 20, 2017

Abstract

Various techniques are disclosed for a system and method to dynamically co-register images captured by two or more separate imaging modules. For example, in one embodiment, a method includes: capturing a thermal image of an object; capturing a visible light (VL) image of the object; determining a plurality of thermal image reference points from the thermal image; determining a plurality of VL image reference points from the VL image; determining a geometric transform based on the plurality of thermal image reference points and the plurality of VL image reference points; and applying the geometric transform to the thermal image or the VL image to align the thermal image and the VL image. In another embodiment, an imaging system comprises two or more separate imaging modules and one or more processors configured to perform the method above to dynamically co-register images.

First claim

What is claimed is:

1. An imaging system, comprising:
a thermal imaging device comprising a thermal imaging module configured to capture a thermal image of an object;
a visible light(VL) imaging device comprising a VL imaging module configured to capture a VL image of the object; and
one or more processors communicatively coupled to the thermal imaging module and the VL imaging module, the one or more processors configured to:
determine a thermal edge image based on the thermal image, the thermal edge image representing edges and/or contours captured in the thermal image,
determine a VL edge image based on the VL image, the VL edge image representing edges and/or contours captured in the VL image,
determine, based on the VL edge image and the thermal edge image, a horizontal translation and a vertical translation from a range of horizontal and vertical translations, such that a correlation between the VL edge image and the thermal edge image is highest when the VL edge image or the thermal edge is translated according to the horizontal translation and the vertical translation, and
apply the horizontal translation and the vertical translation to the thermal image or the VL image to correct a parallax error between the thermal image and the VL image.

2. The imaging system of claim 1 , wherein the one or more processors are configured to:
determine a plurality of thermal image reference points from the thermal image;
determine a plurality of VL image reference points from the VL image,
determine a geometric transform based on the plurality of thermal image reference points and the plurality of VL image reference points, such that the geometric transform at least approximately maps the plurality of thermal image reference points to the plurality of VL image reference points or at least approximately maps the plurality of VL image reference points to the plurality of thermal image reference points; and
apply the geometric transform to the thermal image or the VL image to align the thermal image and the VL image.

3. The imaging system of claim 2 , wherein:
the VL imaging device includes a mobile device;
the thermal imaging device includes a device attachment configured to releasably attach to the mobile device; and
the one or more processors includes a processor of the mobile device and/or a processor of the device attachment.

4. The imaging system of claim 2 , wherein the device attachment is releasably attachable to the mobile device by engaging a device connector receptacle of the mobile device with a corresponding device connector plug of the device attachment.

5. The imaging system of claim 2 , wherein:
the geometric transform comprises a combination of one or more affine transforms; and
the one or more processors are configured to determine one or more affine transform parameters for the combination of one or more affine transforms to determine the geometric transform.

6. The imaging system of claim 5 , wherein the combination of one or more affine transforms include a translation transform in an X axis, a translation transform in a Y axis, a rotation transform, and a scaling transform.

7. The imaging system of claim 5 , wherein the one or more processors are configured to determine the one or more affine transform parameters by an iterative process.

8. The imaging system of claim 5 , wherein:
the one or more processors are configured to select the one or more affine transform parameters from a range of affine transform parameters to determine the one or more affine transform parameters; and
the combination of one or more affine transforms with the selected one or more affine transform parameters maps the plurality of thermal image reference points to the plurality of VL image reference points or maps the plurality of VL image reference points to the plurality of thermal image reference points, such that a mean square error of the distances between the plurality of VL image reference points and the corresponding plurality of thermal image reference points is minimized over the range of affine transform parameters.

9. The imaging system of claim 2 , wherein the one or more processors are configured to:
detect peak corner points from the object captured in the thermal image as the plurality of thermal image reference points; and
detect peak corner points from the object captured in the VL image as the plurality of VL image reference points.

10. The imaging system of claim 1 , wherein the one or more processors are configured to determine a correlation between the VL edge image and the thermal edge image for each of the range of horizontal and vertical translations applied to the VL edge image or the thermal edge image, wherein the correlation is determined by a two-dimensional (2D) dot product of the VL edge image and the thermal edge image.

11. The imaging system of claim 1 , wherein the one or more processors are configured to determine the thermal edge image and the VL edge image at least by detecting the edges and/or contours from the thermal image and the VL image.

12. A method comprising:
capturing a thermal image of an object;
capturing a visible light (VL) image of the object;
determining a thermal edge image based on the thermal image, the thermal edge image representing edges and/or contours captured in the thermal image;
determining a VL edge image based on the VL image, the VL edge image representing edges and/or contours captured in the VL image;
determining, based on the VL edge image and the thermal edge image, a horizontal translation and a vertical translation from a range of horizontal and vertical translations, such that a correlation between the VL edge image and the thermal edge image is highest when the VL edge image or the thermal edge is translated according to the horizontal translation and the vertical translation; and
applying the horizontal translation and the vertical translation to the thermal image or the VL image to correct a parallax error between the thermal image and the VL image.

13. The method of claim 12 , comprising:
determining a plurality of thermal image reference points from the thermal image;
determining a plurality of VL image reference points from the VL image;
determining a geometric transform based on the plurality of thermal image reference points and the plurality of VL image reference points, such that the geometric transform at least approximately maps the plurality of thermal image reference points to the plurality of VL image reference points or at least approximately maps the plurality of VL image reference points to the plurality of thermal image reference points; and
applying the geometric transform to the thermal image or the VL image to align the thermal image and the VL image.

14. The method of claim 13 , wherein:
the geometric transform comprises a combination of one or more affine transforms; and
the determining of the geometric transform comprises determining one or more affine transform parameters for the combination of one or more affine transforms.

15. The method of claim 14 , wherein the combination of one or more affine transforms include a translation transform in an X axis, a translation transform in a Y axis, a rotation transform, and a scaling transform.

16. The method of claim 14 , wherein the determining of the one or more affine transform parameters is by an iterative process.

17. The method of claim 14 , wherein:
the determining of the one or more affine transform parameters comprises selecting the one or more affine transform parameters from a range of affine

Assignees

• Flir Systems

Inventors

• Grant Gregory
• Högasten Nicholas
• Mcnally Scott
• Cagley Richard

Classifications

• H04N23/11

  for generating image signals from visible and infrared light wavelengths · CPC title

• G06T7/33 Primary

  using feature-based methods · CPC title

• G06T2207/10004

  Still image; Photographic image · CPC title

• G06T2207/20221

  Image fusion; Image merging · CPC title

• H04N23/45

  for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images · CPC title