System and apparatus for co-registration and correlation between multi-modal imagery and method for same

Patent metadata

Field	Value
Publication number	US-2018302564-A1
Application number	US-201815952909-A
Country	US
Kind code	A1
Filing date	Apr 13, 2018
Priority date	Apr 14, 2017
Publication date	Oct 18, 2018
Grant date	—

Abstract

The present disclosure provides an image capturing device that captures images of a first sensor that includes a first imaging modality, a second sensor that includes a first imaging modality and a third sensor that includes a second imaging modality. A controller connected with the first sensor, the second sensor and the third sensor, wherein the controller registers an image captured by the first sensor or the second sensor to an image captured by the third sensor.

First claim

We claim:

1 . An image capturing device that captures images, comprising:
a first sensor includes a first imaging modality; a second sensor includes the first imaging modality; a third sensor includes a second imaging modality; and a controller connected with the first sensor, the second sensor and the third sensor, wherein the controller registers an image captured by the first sensor or the second sensor to an image captured by the third sensor.

2 . The image capturing device of claim 1 , wherein the image registration is based on the depth information calculated based on information from the first sensor and the second sensor.

3 . The image capturing device of claim 1 , wherein the first imaging modality comprises color imaging.

4 . The image capturing device of claim 1 , wherein the first imaging modality comprises near-infrared imaging.

5 . The image capturing device of claim 1 , wherein the second imaging modality comprises thermal imaging.

6 . The image capturing device of claim 1 , wherein the second imaging modality comprises at least of one of terahertz imaging, hyperspectral imaging, fluorescence imaging, narrow band imaging and oxygen saturation imaging.

7 . The image capturing device of claim 1 , further comprising a light source to illuminate a subject captured by at least one of the first sensor, the second sensor and the third sensor.

8 . The image capturing device of claim 7 , wherein the light source further comprises at least one light emitting diode.

9 . The image capturing device of claim 1 , further comprising at least one actuator connected with at least one of the first sensor, the second sensor and the third sensor, for adjusting a position of at least one of the first sensor, the second sensor and the third sensor.

10 . The image capturing device of claim 1 , further comprising an unmanned aerial vehicle.

11 . The image capturing device of claim 1 , wherein the first sensor, the second sensor and the third sensor provide information to control a vehicle.

12 . The image capturing device of claim 1 , further comprising a display to display an output of the registered image.

13 . The image capturing device of claim 1 , wherein the first sensor, the second sensor and the third sensor are housed in at least one of a robot, a motor vehicle, a toy, a baby monitor and a head-mounted display.

14 . The image capturing device of claim 1 , further comprising a commutation interface to remotely communicate data from the first sensor, the second sensor and the third sensor.

15 . The image capturing device of claim 1 , where the third sensor having the second imaging modality is positioned between or on top of the first sensor and the second sensor having the first imaging modality.

16 . The image capturing device of claim 1 , further comprising a filter positioned in front of at least one of the first sensor, the second sensor and the third sensor and a subject.

17 . An image capturing device for use with a device having a controller and a communication interface, comprising:
a first sensor includes a first imaging modality; a second sensor includes the first imaging modality; a third sensor includes a second imaging modality; and a communication interface adapted to communicate with the communication interface of the device to send an image captured by at least one of the first sensor and the second sensor, and an image captured by the third sensor, to the controller; wherein the controller of the device registers the first image captured by the first sensor to the second image captured by the third sensor.

18 . The image capturing device of claim 17 , wherein the registration is based on the depth information calculated based on the first sensor and the second sensor.

19 . The image capturing device of claim 17 , wherein the first imaging modality comprises color imaging.

20 . The image capturing device of claim 17 , wherein the first imaging modality comprises near-infrared imaging.

21 . The image capturing device of claim 17 , wherein the second imaging modality comprises thermal imaging.

22 . The image capturing device of claim 17 , wherein the second imaging modality comprises at least of one of terahertz imaging, hyperspectral imaging, fluorescence imaging, narrow band imaging and oxygen saturation imaging.

23 . The image capturing device of claim 17 , further comprising a light source to illuminate a subject captured by at least one of the first sensor, the second sensor and the third sensor.

24 . The image capturing device of claim 23 , wherein the light source further comprises at least one light emitting diode.

25 . The image capturing device of claim 17 , further comprising at least one actuator connected with at least one of the first sensor, the second sensor and the third sensor, for adjusting a position of at least one of the first sensor, the second sensor and the third sensor.

26 . The image capturing device of claim 17 , further comprising an unmanned aerial vehicle.

27 . The image capturing device of claim 17 , further comprising a display to display an output of the registered image.

28 . The image capturing device of claim 17 , wherein the first sensor, the second sensor and the third sensor are housed in at least one of a robot, a vehicle, a toy, a baby monitor and a head-mounted display.

29 . The image capturing device of claim 17 , where the third sensor having the second imaging modality is positioned between or on top of the first sensor and the second sensor having the first imaging modality.

30 . The image capturing device of claim 17 , further comprising a filter positioned in front of at least one of the first sensor, the second sensor and the third sensor and a subject.

31 . A method of registering images of a first imaging modality to images of a second imaging modality, comprising:
capturing a first image using a first sensor that includes a first imaging modality; capturing a second image using a second sensor that includes the first imaging modality; capturing a third image using a third sensor that includes a second imaging modality; determining a first depth map for at least one pixel of the first image based on the first image and the second image; and registering the first image or the second image to the third image based on the first depth map.

32 . The method of claim 31 , further comprising displaying a composite image of the first image registered to the third image.

33 . The method of claim 31 , wherein the first imaging modality comprises color imaging.

34 . The method of claim 31 , wherein the first imaging modality comprises near-infrared imaging.

35 . The method of claim 31 , wherein the second imaging modality comprises at least of one of thermal imaging, terahertz imaging, hyperspectral imaging, fluorescence imaging, narrow band imaging and oxygen saturation imaging.

36 . The method of claim 31 , further comprising:
calculating a second depth map for at least one pixel of the second image based on the first image and the second image; and registering the second image to the third image based on the second depth map.

37 . The

Assignees

• Liu Yang
• Askari Karchegani Maziyar

Inventors

• Liu Yang
• Askari Karchegani Maziyar

Classifications

• H04N23/667 Primary

  Camera operation mode switching, e.g. between still and video, sport and normal or high- and low-resolution modes · CPC title

• H04N23/45

  for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images · CPC title

• H04N23/11

  for generating image signals from visible and infrared light wavelengths · CPC title

• H04N23/56

  provided with illuminating means · CPC title

• H04N23/66

  Remote control of cameras or camera parts, e.g. by remote control devices · CPC title