Image processing device, imaging device, image processing method, and program

Patent metadata

Field	Value
Publication number	US-2016140697-A1
Application number	US-201615003849-A
Country	US
Kind code	A1
Filing date	Jan 22, 2016
Priority date	Aug 2, 2013
Publication date	May 19, 2016
Grant date	—

Abstract

Disclosed are an image processing device, an imaging device, an image processing method, and a program capable of acquiring a moving image with excellent image quality while maintaining continuity of a restoration process between frames even if there is a rapid change of a photographing environment in a moving image. An image processing device includes a restoration control processing unit 36 which subjects a moving image including a plurality of frames acquired by photographing using an optical system to a restoration process based on a point spread function of the optical system to acquire recovered image data. The restoration control processing unit 36 controls the restoration process for a frame to be processed among a plurality of frames based on imaging information of a reference frame including a frame after the frame to be processed in a time series.

First claim

What is claimed is:

1 . An image processing device comprising:
a restoration control processing unit which subjects a moving image including a plurality of frames acquired by photographing using an optical system to a restoration process based on a point spread function of the optical system to acquire recovered image data, wherein the restoration control processing unit controls the restoration process for a frame to be processed among the plurality of frames based on imaging information of a reference frame including a frame after the frame to be processed in a time series, and wherein the imaging information includes photographing condition information of the frame to be processed and the reference frame.

2 . The image processing device according to claim 1 ,
wherein the reference frame includes a frame before the frame to be processed in a time series.

3 . The image processing device according to claim 2 wherein the reference frame includes a frame immediately before and a frame immediately after the frame to be processed in a time series.

4 . The image processing device according to claim 1 ,
wherein the restoration control processing unit controls the restoration process for the frame to be processed based on the imaging information of the reference frame and the imaging information of the frame to be processed.

5 . The image processing device according to claim 1 ,
wherein the photographing condition information includes at least one of a diaphragm value and a zoom magnification.

6 . The image processing device according to claim 1 ,
wherein the restoration control processing unit performs the restoration process based on the most frequent photographing condition among the photographing condition information of the frame to be processed and the reference frame.

7 . The image processing device according to claim 1 ,
wherein the restoration process includes a filtering process using a restoration filter based on the point spread function, and the restoration control processing unit performs a filtering process using, as a restoration filter, a filter calculated based on a restoration filter corresponding to the photographing condition information of the frame to be processed and a restoration filter corresponding to the photographing condition information of the reference frame for the frame to be processed to perform the restoration process for the frame to be processed.

8 . The image processing device according to claim 7 ,
wherein the restoration control processing unit performs a filtering process using, as a restoration filter, a filter calculated from a weighted average of a restoration filter corresponding to the photographing condition information of the frame to be processed and a restoration filter corresponding to the photographing condition information of the reference frame for the frame to be processed to perform the restoration process for the frame to be processed.

9 . The image processing device according to claim 7 ,
wherein the restoration control processing unit sets image data calculated from a weighted average of image data calculated by applying a restoration filter corresponding to the photographing condition information of the frame to be processed to the frame to be processed and image data calculated by applying a restoration filter corresponding to the photographing condition information of the reference frame to the reference frame as the recovered image data of the frame to be processed.

10 . The image processing device according to claim 7 , further comprising:
an object detection unit which detects motion of an object of the moving image, wherein, in case where the motion amount of the object in the frame to be processed and the reference frame detected by the object detection unit is equal to or greater than a threshold value, the restoration control processing unit performs a filtering process using, as a restoration filter, a filter calculated from a weighted average of a restoration filter corresponding to the photographing condition information of the frame to be processed and a restoration filter corresponding to the photographing condition information of the reference frame for the frame to be processed to perform the restoration process for the frame to be processed, and in case where the motion amount of the object in the frame to be processed and the reference frame detected by the object detection unit is less than the threshold value, the restoration control processing unit sets image data calculated from a weighted average of image data calculated by applying a restoration filter corresponding to the photographing condition information of the frame to be processed to the frame to be processed and image data calculated by applying a restoration filter corresponding to the photographing condition information of the reference frame to the reference frame as the recovered image data of the frame to be processed.

11 . The image processing device according to claim 1 ,
wherein the imaging information includes image information of the frame to be processed and the reference frame.

12 . The image processing device according to claim 11 ,
wherein the image information includes at least one of information regarding whether or not a saturated pixel is included in the frame to be processed and the reference frame and information regarding whether or not a defocus region is included in the frame to be processed and the reference frame.

13 . The image processing device according to claim 11 ,
wherein the image information includes information regarding whether or not a saturated pixel is included in the frame to be processed and the reference frame, and the restoration control processing unit determines whether or not a saturated pixel is included in the reference frame, and in case where it is determined that the saturated pixel is included in the reference frame, adjusts the restoration intensity of the restoration process for the frame to be processed based on the imaging information of the reference frame where the saturated pixel is included.

14 . The image processing device according to claim 13 ,
wherein the restoration control processing unit analyzes a blinking state of a light source in the reference frame, and adjusts the restoration intensity of the restoration process for the frame to be processed based on the blinking state of the light source.

15 . The image processing device according to claim 11 ,
wherein the image information includes information regarding whether or not a defocus region is included in the frame to be processed and the reference frame, and the restoration control processing unit determines whether or not a defocus region is included in the reference frame, and in case where it is determined that the defocus region is included in the reference frame, adjusts the restoration intensity of the restoration process for the frame to be processed based on the imaging information of the reference frame where the defocus region is included.

16 . The image processing device according to claim 1 , further comprising:
a moving body detection unit which detects a moving body in the moving image, wherein, in case where the moving body is detected in the frame to be processed and the reference frame by the moving body detection unit, the restoration control processing unit subjects the frame to be processed to the restoration process based on the point spread function reflecting phase characteristics based on motion of the moving body.

17 . The image

Assignees

• Fujifilm Corp

Inventors

• Sugimoto Masahiko
• Hayashi Kenkichi
• Naruse Yousuke
• Irie Kosuke

Classifications

• G06T5/50

  using two or more images, e.g. averaging or subtraction · CPC title

• G06T2207/20201

  Motion blur correction · CPC title

• G06T2207/10016

  Video; Image sequence · CPC title

• G06T2207/20192

  Edge enhancement; Edge preservation · CPC title

• G06T5/20

  using local operators · CPC title