Image processing device, imaging device, image processing method, and image processing program

Patent metadata

Field	Value
Publication number	US-10395347-B2
Application number	US-201715789015-A
Country	US
Kind code	B2
Filing date	Oct 20, 2017
Priority date	Apr 23, 2015
Publication date	Aug 27, 2019
Grant date	Aug 27, 2019

Abstract

In a case where IR data of a near-infrared light image is data in which a visible light component and a near-infrared light component coexist, a point image restoration process is performed on the IR data using a first point image restoration filter based on a first point spread function for visible light of the optical system and a second point image restoration filter based on a second point spread function for near-infrared light of the optical system. An appropriate point image restoration process is performed on the IR data that is captured in a time zone of twilight or dawn by performing weighted averaging on a point image restoration process using the first point image restoration filter and a point image restoration process using the second point image restoration filter using a first gain α and a second gain β according to a light amount ratio between visible light and near-infrared light at the time of capturing the IR data.

First claim

What is claimed is:

1. An image processing device, comprising:
an image acquisition unit that acquires image data including a near-infrared light image captured with sensitivity to a visible light wavelength band and a near-infrared light wavelength band using an optical system;
a point image restoration processing unit that performs a point image restoration process on the acquired image data using a first point image restoration filter based on a first point spread function for visible light of the optical system and a second point image restoration filter based on a second point spread function for near-infrared light of the optical system; and
a restoration rate control unit that controls the point image restoration processing unit to adjust a first restoration rate in the point image restoration process using the first point image restoration filter and a second restoration rate in the point image restoration process using the second point image restoration filter for the acquired image data, wherein:
the restoration rate control unit includes a light amount ratio detection unit that detects a light amount ratio between a first light amount by visible light and a second light amount by near-infrared light at the time of capturing the near-infrared light image; and
the restoration rate control unit adjusts the first restoration rate and the second restoration rate according to the detected light amount ratio.

2. The image processing device according to claim 1 , wherein:
the point image restoration processing unit applies the first point image restoration filter and the second point image restoration filter to the acquired image data to generate first increment or decrement data and second increment or decrement data, and adds the first increment or decrement data and the second increment or decrement data that have been generated to the image data; and
the restoration rate control unit adjusts a first gain for the first increment or decrement data and a second gain for the second increment or decrement data according to the light amount ratio detected by the light amount ratio detection unit to adjust the first restoration rate and the second restoration rate.

3. The image processing device according to claim 2 , wherein the restoration rate control unit acquires a total gain based on the first gain and the second gain, and adjusts a ratio between the first gain and the second gain in the acquired total gain according to the light amount ratio detected by the light amount ratio detection unit.

4. The image processing device according to claim 1 , wherein:
the image data acquired by the image acquisition unit is continuously captured moving image data; and
the light amount ratio detection unit measures the amount of light in an imaging period of a plurality of frames of the moving image data, and detects the light amount ratio between the first amount of light and the second amount of light on the basis of the measured amount of light.

5. The image processing device according to claim 1 , wherein:
the image acquisition unit further acquires image data indicating a visible light image captured with sensitivity to a visible light wavelength band using the optical system; and
the point image restoration processing unit performs the point image restoration process on the image data indicating the visible light image using the first point image restoration filter based on the first point spread function for the visible light of the optical system.

6. The image processing device according to claim 5 , wherein:
the image data indicating the visible light image includes first color data, and second color data of two or more colors having a contribution rate for obtaining luminance data lower than that of the first color data; and
the point image restoration processing unit performs the point image restoration process on the luminance data generated from the image data indicating the visible light image using the first point image restoration filter corresponding to the luminance data.

7. The image processing device according to claim 5 , wherein:
the image data indicating the visible light image includes first color data, and second color data of two or more colors having a contribution rate for obtaining luminance data lower than that of the first color data; and
the point image restoration processing unit performs the point image restoration process on the first color data and each of pieces of the second color data of two or more colors using the first point image restoration filter corresponding to each of the first color data and each of pieces of the second color data of two or more colors.

8. The image processing device according to claim 1 , wherein in a case where the acquired image data is image data of only a near-infrared light component, the point image restoration processing unit performs only a point image restoration process on the image data of only the near-infrared light component using a second point image restoration filter based on a second point spread function for the near-infrared light of the optical system.

9. An image processing device, comprising:
an image acquisition unit that acquires image data including the near-infrared light image captured with sensitivity to a visible light wavelength band and a near-infrared light wavelength band using an optical system; and
a point image restoration processing unit that performs a point image restoration process on the acquired image data using a point image restoration filter based on a point spread function for visible light and near-infrared light of the optical system, wherein:
the point image restoration processing unit includes a light amount ratio detection unit that detects a light amount ratio between a first amount of light by the visible light and a second amount of light by the near-infrared light at the time of capturing the near-infrared light image when performing the point image restoration process using the point image restoration filter; and
the point image restoration processing unit performs the point image restoration process using the point image restoration filter based on the point spread function according to the detected light amount ratio.

10. The image processing device according to claim 9 , wherein:
the point image restoration processing unit includes:
a point spread function generation unit that generates the point spread function for visible light and near-infrared light of the optical system obtained by performing weighted averaging on a first point spread function for visible light of the optical system and a second point spread function for near-infrared light of the optical system according to the light amount ratio detected by the light amount ratio detection unit; and
a point image restoration filter generation unit that generates the point image restoration filter on the basis of the generated point spread function; and

the point image restoration processing unit performs the point image restoration process using the generated point image restoration filter.

11. The image processing device according to claim 9 , wherein:
the point image restoration processing unit includes:
a point spread function storage unit that stores a plurality of point spread functions corresponding to the light amount ratio detected by the light amount ratio detection unit; and
a point image restoration filter generation unit that reads the point spread function corresponding to the light amount ratio detected by the light amount ratio detection unit from the point spread function storage unit, and generates the point image restoration filter from the read point spread function; and

the p

Assignees

• Fujifilm Corp

Inventors

• Tanaka Junichi
• Hayashi Kenkichi
• Naruse Yousuke

Classifications

• H04N23/667

  Camera operation mode switching, e.g. between still and video, sport and normal or high- and low-resolution modes · CPC title

• H04N25/131

  including elements passing infrared wavelengths · CPC title

• H04N25/135

  based on four or more different wavelength filter elements · CPC title

• H04N23/11

  for generating image signals from visible and infrared light wavelengths · CPC title

• H04N23/843

  Demosaicing, e.g. interpolating colour pixel values · CPC title