Multiscale depth estimation using depth from defocus

Patent metadata

Field	Value
Publication number	US-2016182893-A1
Application number	US-201514972477-A
Country	US
Kind code	A1
Filing date	Dec 17, 2015
Priority date	Dec 22, 2014
Publication date	Jun 23, 2016
Grant date	—

Abstract

To extend the working range of depth from defocus (DFD) particularly on small depth of field (DoF) images, DFD is performed on an image pair at multiple spatial resolutions and the depth estimates are then combined. Specific implementations construct a Gaussian pyramid for each image of an image pair, perform DFD on the corresponding pair of images at each level of the two image pyramids, convert DFD depth scores to physical depth values using calibration curves generated for each level, and combine the depth values from all levels in a coarse-to-fine manner to obtain a final depth map that covers the entire depth range of the scene.

First claim

1 . A method of generating a merged physical depth map from a pair of images of a scene captured by an image capture device using different capture parameters, said method comprising:
generating, from the captured pair of images, at least one pair of downsampled images at a corresponding spatial resolution; and for each pair of images:
forming a relative depth map from the pair of images;
accessing calibration data of the image capture device associated with the pair of images, the calibration data relating relative depth measurements with physical depth and having a defined working range where a single relative depth measurement relates to a single physical depth; and
determining a physical depth map for the pair of the images from the formed relative depth map and the accessed calibration data; and
generating a merged physical depth map by merging, in order of increasing spatial resolution, depth values within the defined working range associated with each of the determined physical depth maps.

2 . A method according to claim 1 , wherein generating the merged physical depth map uses confidence measures associated with the relative depth estimates to weight the physical depth values from corresponding spatial resolutions when merging the physical depth values.

3 . A method according to claim 1 , wherein the generating the merged physical depth map refines the physical depth map at each spatial resolution by:
upsampling a depth map of the previous coarser resolution and using the upsampled depth map as an initial refined depth map; forming a current refined depth map by minimising an energy function that penalises depth values that, at least one of:
(i) deviate from the physical depth measurements of the current spatial resolution; and
(ii) produce large local depth gradients .

4 . A method according to claim 3 wherein for (i), the energy function operates such that a data penalty increases with the associated confidence measure of the depth measurements.

5 . A method according to claim 3 wherein the penalty of (ii) is reduced at pixels where a large local depth gradient is matched by a large local intensity gradient in the associated image.

6 . A method according to claim 3 , further comprising modifying depth boundaries of the refined depth map by:
(i) locating pixels that are potentially lying on a depth boundary; (ii) dilating regions of said located pixels; and (iii) performing edge-preserved filtering on said dilated regions of said located pixels to modify the corresponding physical depth values and thereby the corresponding depth boundary.

7 . A method according to claim 6 , wherein the edge-preserved filtering comprises joint bilateral filtering.

8 . A method according to claim 6 wherein step (i) comprises locating pixels with an associated image and whose depth gradient exceeds a pre-defined threshold to thereby define a depth boundary.

9 . A method according to claim 6 wherein step (i) comprises identifying pixels whose depth gradient exceeds a pre-defined threshold and lie on an edge on the associated image.

10 . A non-transitory computer readable storage medium having a program recorded thereon, the program being executable by a processor to generate merged physical depth map from a pair of images of a scene captured by an image capture device using different capture parameters, said program comprising:
code for generating, from the captured pair of images, at least one pair of downsampled images at a corresponding spatial resolution; and code, operative for each pair of images, to:
form a relative depth map from the pair of images;
access calibration data of the image capture device associated with the pair of images, the calibration data relating relative depth measurements with physical depth and having a defined working range where a single relative depth measurement relates to a single physical depth; and
determine a physical depth map for the pair of the images from the formed relative depth map and the accessed calibration data; and
code for generating a merged physical depth map by merging, in order of increasing spatial resolution, depth values within the defined working range associated with each of the determined physical depth maps.

11 . A non-transitory computer readable storage medium according to claim 10 , wherein the code for generating the merged physical depth map uses confidence measures associated with the relative depth estimates to weight the physical depth values from corresponding spatial resolutions when merging the physical depth values.

12 . A non-transitory computer readable storage medium according to claim 10 , wherein the code for generating the merged physical depth map refines the physical depth map at each spatial resolution by:
upsampling a depth map of the previous coarser resolution and using the upsampled depth map as an initial refined depth map; forming a current refined depth map by minimising an energy function that penalises depth values that, at least one of:
(i) deviate from the physical depth measurements of the current spatial resolution ; and
(ii) produce large local depth gradients .

13 . A non-transitory computer readable storage medium according to claim 12 wherein for (i), the energy function operates such that a data penalty increases with the associated confidence measure of the depth measurements, and the penalty of (ii) is reduced at pixels where a large local depth gradient is matched by a large local intensity gradient in the associated image.

14 . A non-transitory computer readable storage medium according to claim 11 , further comprising code for modifying depth boundaries of the refined depth map by:
(i) locating pixels that are potentially lying on a depth boundary; (ii) dilating regions of said located pixels; and (iii) performing edge-preseved filtering on said dilated regions of said located pixels to modify the corresponding physical depth values and thereby the corresponding depth boundary.

15 . A non-transitory computer readable storage medium according to claim 14 wherein step (i) comprises locating pixels with an associated image and whose depth gradient exceeds a pre-defined threshold to thereby define a depth boundary, and the edge-preserved filtering comprises joint bilateral filtering.

16 . A non-transitory computer readable storage medium according to claim 14 wherein step (i) comprises identifying pixels whose depth gradient exceeds a pre-defined threshold and lie on an edge on the associated image, and the edge-preserved filtering comprises joint bilateral filtering.

17 . Apparatus comprising a processor and a memory, the memory storing a program executable by the processor for generating a merged physical depth map from a pair of images of a scene captured by an image capture device using different capture parameters, the program comprising:
code for generating, from the captured pair of images, at least one pair of downsampled images at a corresponding spatial resolution; and code, operative for each pair of images, to :
form a relative depth map from the pair of images;
access calibration data of the image capture device associated with the pair of images, the calibration data relating relative depth measurements with physical depth and having a defined working range where a single relative depth measurement relates to a single physical depth; and
determine a physical depth map for the pair of the images from the formed relative depth map and the accessed calibration data; and
code for generating a merged physical dep

Assignees

• Canon Kk

Inventors

• Wan Ernest Yiu Cheong

Classifications

• G06T7/0075

  Physics · mapped topic

• H04N13/0207

  Electricity · mapped topic

• H04N13/0246

  Electricity · mapped topic

• H04N13/0282

  Electricity · mapped topic

• G06T2207/20028

  Bilateral filtering · CPC title