Image processing device having depth map generating unit, image processing method and non-transitory computer redable recording medium

Patent metadata

Field	Value
Publication number	US-9508173-B2
Application number	US-201414526524-A
Country	US
Kind code	B2
Filing date	Oct 29, 2014
Priority date	Oct 30, 2013
Publication date	Nov 29, 2016
Grant date	Nov 29, 2016

Abstract

An object of the present invention is to provide an image processing device and the like that can generate a composite image in a desired focusing condition. In a smartphone 1 , an edge detecting section 107 detects an edge as a feature from a plurality of input images taken with different focusing distances, and detects the intensity of the edge as a feature value. A depth estimating section 111 then estimates the depth of a target pixel, which is information representing which of the plurality of input images is in focus at the target pixel, by using the edge intensity detected by the edge detecting section 107 . A depth map generating section 113 then generates a depth map based on the estimation results by the depth estimating section 111.

First claim

The invention claimed is:

1. An image processing device, comprising:
a contrast correcting unit configured to correct contrast of a plurality of input images;
a feature detecting unit configured to detect a predetermined feature from the plurality of input images that are corrected by the contrast correcting unit;
a depth map generating unit configured to generate a depth map based on a detection result from the feature detecting unit,
wherein the feature detecting unit comprises a feature value detecting unit configured to detect a predetermined feature value from the plurality of input images,
the image processing device further comprises a depth estimating unit configured to estimate a depth of a target pixel by means of the feature value detected by the feature value detecting unit, the depth at the target pixel being information representing which of the plurality of input images is in focus at the target pixel, and
the depth map generating unit generates the depth map based on an estimation result from the depth estimating unit.

2. The image processing device according to claim 1 ,
wherein the depth estimating unit performs a predetermined proper solution processing on the target pixel by using the respective feature values of the plurality of input images detected by the feature value detecting unit so as to determine an input image that is in focus at the target pixel from among the plurality of input images, in which the proper solution processing uses a first parameter that is determined by the feature value of the target pixel, and a second parameter that is a penalty imposed when a first input image that is in focus at the target pixel is different from a second input image that is in focus at a neighboring pixel of the target pixel.

3. The image processing device according to claim 2 , further comprising:
a motion blur area detecting unit configured to detect a motion blur area in the plurality of input images,
wherein in the proper solution processing, at least one of the first parameter and the second parameter is set to a different parameter value for the motion blur area detected by the motion blur area from parameter values for an area other than the motion blur area.

4. The image processing device according to claim 1 , further comprising:
a focus separation determination unit configured to determine as to whether focus is correctly separated in the plurality of input images,
wherein the depth estimating unit estimates the depth of the target pixel using input images for which a determination result from the focus separation determination unit is positive.

5. The image processing device according to claim 1 ,
wherein the feature value detecting unit detects the feature value of the plurality of input images or a plurality of reduced input images that are obtained by reducing the plurality of input images,
the depth map generating unit generates a reduced depth map based on the estimation result from the depth estimating unit, and
the image processing device further comprises:
a reconstructing unit configured to reconstruct a reconstructed depth map from the reduced depth map; and
a correcting unit configured to generate a corrected depth map from the reconstructed depth map reconstructed by the reconstructing unit by means of a predetermined corrector image.

6. The image processing device according to claim 1 , further comprising:
an input unit configured to receive a user operation,
wherein the depth estimating unit comprises a depth re-estimating unit configured to re-estimate the depth of a pixel included in a designated area designated by the user operation, and
the depth map generating unit comprises a depth map regenerating unit configured to regenerate the depth map based on a re-estimation result from the depth re-estimating unit.

7. The image processing device according to claim 2 , further comprising:
an input unit configured to receive a user operation,
wherein the depth estimating unit comprises a depth re-estimating unit configured to re-estimate the depth of a pixel included in a designated area designated by the user operation,
the depth map generating unit comprises a depth map regenerating unit configured to regenerate the depth map based on a re-estimation result from the depth re-estimating unit, and
the depth re-estimating unit performs the proper solution processing to the pixel in the designated area so as to re-estimate the depth, in which at least one of the first parameter and the second parameter is set to a different parameter value from parameter values that are originally used in depth estimation by the depth estimating unit.

8. An image processing device, comprising:
a feature detecting unit configured to detect a predetermined feature from a plurality of input images;
a depth map generating unit configured to generate a depth map based on a detection result from the feature detecting unit,
wherein the feature detecting unit comprises a feature point detecting unit configured to detect feature points from the plurality of input images, in which the plurality of input images are a plurality of photographed images that have parallax by being taken from different viewpoints,
the image processing device further comprises:
a relative viewpoint estimating unit configured to estimate relative positional relation between respective viewpoints of the plurality of photographed images based on the feature points detected by the feature point detecting unit; and
a pixel position estimating unit configured to perform a predetermined estimating calculation so as to estimate three-dimensional positions of pixels of the plurality of input images, and
the depth map generating unit generates the depth map based on an estimation result from the relative viewpoint estimating unit and an estimation result from the pixel position estimating unit.

9. The image processing device according to claim 8 ,
wherein the relative viewpoint estimating unit comprises:
a feature point tracking unit configured to track the feature points detected by the feature point detecting unit; and
a base input image extracting unit configured to extract an input image from among the plurality of input images as an base input image to be used as a reference point based on a tracking result from the feature point tracking unit, and
the relative viewpoint tracking unit estimates the relative positional relation of the respective viewpoints of the plurality of photographed images based on the feature points detected by the feature point detecting unit from the base input image extracted by the base input image extracting unit.

10. The image processing device according to claim 8 , further comprising:
a display unit;
a display controlling unit configured to control the display unit to display a guide for showing a user a moving manner of the device that causes parallax between the plurality of photographed images; and
a movement parameter detecting unit configured to detect a movement parameter of the device,
wherein the display controlling unit changes an appearance of the guide based on a detection result from the movement parameter detecting unit.

11. An image processing device, comprising:
a feature detecting unit configured to detect a predetermined feature from a plurality of input images;
a depth map generating unit configured to generate a depth map based on a detection result from the feature detecting unit;
a display unit;
an input unit configured to receive a user operation; and
a composite image generating unit configured to generate a composite image based on the depth map according to the user operation on the input unit, in w

Assignees

• Morpho Inc

Inventors

• Miura Takeshi
• Sawai shogo
• Kuroda Yasuhiro
• Koyama Hiroshi
• Hiramoto Kazuhiro
• Ishiguro Kosuke
• Ono Ryo

Classifications

• G06F3/04845

  for image manipulation, e.g. dragging, rotation, expansion or change of colour · CPC title

• G06T7/0069

  Physics · mapped topic

• G06T5/50

  using two or more images, e.g. averaging or subtraction · CPC title

• G06T11/60 Primary

  Creating or editing images; Combining images with text · CPC title

• G06F3/04883

  for inputting data by handwriting, e.g. gesture or text · CPC title