Systems and methods for dynamic registration of multimodal images

Patent metadata

Field	Value
Publication number	US-2016196653-A1
Application number	US-201514986571-A
Country	US
Kind code	A1
Filing date	Dec 31, 2015
Priority date	Dec 31, 2014
Publication date	Jul 7, 2016
Grant date	—

Abstract

Various techniques are disclosed for a system and method to dynamically co-register images captured by two or more separate imaging modules. For example, in one embodiment, a method includes: capturing a thermal image of an object; capturing a visible light (VL) image of the object; determining a plurality of thermal image reference points from the thermal image; determining a plurality of VL image reference points from the VL image; determining a geometric transform based on the plurality of thermal image reference points and the plurality of VL image reference points; and applying the geometric transform to the thermal image or the VL image to align the thermal image and the VL image. In another embodiment, an imaging system comprises two or more separate imaging modules and one or more processors configured to perform the method above to dynamically co-register images.

First claim

What is claimed is:

1 . An imaging system, comprising:
a thermal imaging device comprising a thermal imaging module configured to capture a thermal image of an object; a visible light (VL) imaging device comprising a VL imaging module configured to capture a VL image of the object; and one or more processors communicatively coupled to the thermal imaging module and the VL imaging module, the one or more processors configured to:
determine a plurality of thermal image reference points from the thermal image,
determine a plurality of VL image reference points from the VL image,
determine a geometric transform based on the plurality of thermal image reference points and the plurality of VL image reference points, such that the geometric transform at least approximately maps the plurality of thermal image reference points to the plurality of VL image reference points or at least approximately maps the plurality of VL image reference points to the plurality of thermal image reference points, and
apply the geometric transform to the thermal image or the VL image to align the thermal image and the VL image.

2 . The imaging system of claim 1 , wherein:
the VL imaging device includes a mobile device; the thermal imaging device includes a device attachment configured to releasably attach to the mobile device; and the one or more processors includes a processor of the mobile device and/or a processor of the device attachment.

3 . The imaging system of claim 1 , wherein the device attachment is releasably attachable to the mobile device by engaging a device connector receptacle of the mobile device with a corresponding device connector plug of the device attachment.

4 . The imaging system of claim 1 , wherein:
the geometric transform comprises a combination of one or more affine transforms; and the one or more processors are configured to determine one or more affine transform parameters for the combination of one or more affine transforms to determine the geometric transform.

5 . The imaging system of claim 4 , wherein the combination of one or more affine transforms include a translation transform in an X axis, a translation transform in a Y axis, a rotation transform, and a scaling transform.

6 . The imaging system of claim 4 , wherein the one or more processors are configured to determine the one or more affine transform parameters by an iterative process.

7 . The imaging system of claim 4 , wherein:
the one or more processors are configured to select the one or more affine transform parameters from a range of affine transform parameters to determine the one or more affine transform parameters; and the combination of one or more affine transforms with the selected one or more affine transform parameters maps the plurality of thermal image reference points to the plurality of VL image reference points or maps the plurality of VL image reference points to the plurality of thermal image reference points, such that a mean square error of the distances between the plurality of VL image reference points and the corresponding plurality of thermal image reference points is minimized over the range of affine transform parameters.

8 . The imaging system of claim 1 , wherein the one or more processors are configured to:
detect peak corner points from the object captured in the thermal image as the plurality of thermal image reference points; and detect peak corner points from the object captured in the VL image as the plurality of VL image reference points.

9 . The imaging system of claim 1 , wherein the one or more processors are configured to:
receive the aligned thermal image and the aligned VL image; determine a thermal edge image based on the aligned thermal image, the thermal edge image representing edges and/or contours captured in the aligned thermal image; determine a VL edge image based on the aligned VL image, the VL edge image representing edges and/or contours captured in the aligned VL image; determine, based on the VL edge image and the thermal edge image, a horizontal translation and a vertical translation from a range of horizontal and vertical translations, such that a correlation between the VL edge image and the thermal edge image is highest when the VL edge image or the thermal edge is translated according to the horizontal translation and the vertical translation; and apply the horizontal translation and the vertical translation to the aligned thermal image or the aligned VL image to correct a parallax error between the aligned thermal image and the aligned VL image.

10 . The imaging system of claim 9 , wherein the one or more processors are configured to determine a correlation between the VL edge image and the thermal edge image for each of the range of horizontal and vertical translations applied to the VL edge image or the thermal edge image, wherein the correlation is determined by a two-dimensional (2D) dot product of the VL edge image and the thermal edge image.

11 . The imaging system of claim 9 , wherein the one or more processors are configured to determine the thermal edge image and the VL edge image at least by detecting the edges and/or contours from the aligned thermal image and the aligned VL image.

12 . A method comprising:
capturing a thermal image of an object; capturing a visible light (VL) image of the object; determining a plurality of thermal image reference points from the thermal image; determining a plurality of VL image reference points from the VL image; determining a geometric transform based on the plurality of thermal image reference points and the plurality of VL image reference points, such that the geometric transform at least approximately maps the plurality of thermal image reference points to the plurality of VL image reference points or at least approximately maps the plurality of VL image reference points to the plurality of thermal image reference points; and applying the geometric transform to the thermal image or the VL image to align the thermal image and the VL image.

13 . The method of claim 12 , wherein:
the geometric transform comprises a combination of one or more affine transforms; and the determining of the geometric transform comprises determining one or more affine transform parameters for the combination of one or more affine transforms.

14 . The method of claim 13 , wherein the combination of one or more affine transforms include a translation transform in an X axis, a translation transform in a Y axis, a rotation transform, and a scaling transform.

15 . The method of claim 13 , wherein the determining of the one or more affine transform parameters is by an iterative process.

16 . The method of claim 13 , wherein:
the determining of the one or more affine transform parameters comprises selecting the one or more affine transform parameters from a range of affine transform; and the combination of one or more affine transforms with the selected one or more affine transform parameters maps the plurality of thermal image reference points to the plurality of VL image reference points or maps the plurality of VL image reference points to the plurality of thermal image reference points, such that a mean square error of the distances between the plurality of VL image reference points and the corresponding plurality of thermal image reference points is minimized over the range of affine transform parameters.

17 . The method of claim 12 , further comprising:
detecting peak corner points from the object captured in the thermal image as the plurality of therm

Assignees

• Flir Systems

Inventors

• Grant Gregory
• Högasten Nicholas
• Mcnally Scott
• Cagley Richard

Classifications

• G06T2207/10004

  Still image; Photographic image · CPC title

• G06T7/33 Primary

  using feature-based methods · CPC title

• G06T2207/10048

  Infrared image · CPC title

• G06T2207/20221

  Image fusion; Image merging · CPC title

• H04N23/45

  for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images · CPC title