Determining a depth map from images of a scene

Patent metadata

Field	Value
Publication number	US-9836855-B2
Application number	US-201213606822-A
Country	US
Kind code	B2
Filing date	Sep 7, 2012
Priority date	Sep 14, 2011
Publication date	Dec 5, 2017
Grant date	Dec 5, 2017

Abstract

A technique determines a depth measurement associated with a scene captured by an image capture device. The technique receives at least first and second images of the scene, in which the first image is captured using at least one different camera parameter than that of the second image. At least first and second image patches are selected from the first and second images, respectively, the selected patches corresponding to a common part of the scene. The selected image patches are used to determine which of the selected image patches provides a more focused representation of the common part. At least one value is calculated based on a combination of data in the first and second image patches, the combination being dependent on the more focused image patch. The depth measurement of the common part of the scene is determined from the at least one calculated value.

First claim

The invention claimed is:

1. A method for determining a depth measurement associated with a scene captured by an image capture device, the method comprising:
receiving a first image and a second image of the scene captured by the image capture device, the first image being captured using at least one different camera parameter than that of the second image;
selecting a first patch from the first image;
selecting a second patch from the second image to correspond with the first patch such that the first and second patches represent a common part of the scene;
determining which of the first and second patches is more focused;
calculating Fourier transforms for each of the first and second patches; and
calculating a ratio of the Fourier transforms corresponding to the first and second patches to produce an estimate of a complex OTF ratio, with the Fourier transform of the more focused patch as denominator of the ratio, the estimate of the complex OTF ratio representing relative blur of the first and second patches and being a function of distance to the objects in the common part of the scene; and
determining the depth measurement of the common part of the scene from the estimate of the complex OTF ratio.

2. The method according to claim 1 , wherein the estimate of the complex OTF ratio is a mean of the OTF ratio over an area.

3. The method according to claim 1 , wherein the estimate of the complex OTF ratio is a median of the OTF ratio over an area.

4. The method according to claim 1 , wherein the estimate of the complex OTF ratio is a weighted mean of the OTF ratio over an area defined by a weighting function.

5. The method according to claim 4 , wherein the weighting function is zero at spatial frequencies greater than a Nyquist limit of the image capture device.

6. The method according to claim 4 , wherein the weighting function is zero at spatial frequencies determined by selecting a predetermined fraction of pixels with a lower modulus of the Fourier transform of one of the first and second patches.

7. The method according to claim 1 , wherein the ratio of the complex OTFs is determined by:
calculating an arc tangent of ratios of moduli of the Fourier transforms, multiplied by a complex phase equal to a difference of phases of the Fourier transforms, to produce an estimate of the ratio of the complex OTFs, with the complex OTF of the more focused patch as a denominator of the ratio.

8. The method according to claim 1 , wherein determining which of the first and second patches is the more focused patch is done by calculating a variance of each of the first and second patches and selecting the more focused patch as one with a higher variance.

9. The method according to claim 1 , wherein the selecting of the first and second patches comprises aligning the first image to the second image based on content of the images.

10. A method for determining a depth map associated with a scene captured by an image capture device, the method comprising:
receiving a first image and a second image of the scene captured by the image capture device, the first image being captured using at least one different camera parameter than that of the second image;
determining a depth measurement for a plurality of parts of the scene by:
(a) (i) selecting at least a first patch from the first image;
(ii) selecting a second patch from the second image to correspond with the first patch for a current part of the scene such that the first and second patches in the current part represent a common part of the scene;

(b) determining which of the first and second patches is a more focused patch;
(c0) calculating Fourier transforms for each of the first and second patches;
(c) calculating a ratio of the Fourier transforms corresponding to the first and second patches to produce an estimate of a complex OTF ratio, with the Fourier transform of the more focused patch as denominator of the ratio, the estimate of the complex OTF ratio representing relative blur of the first and second patches and being a function of distance to the objects in the common part of the scene;
(d) determining the depth measurement of the current part of the scene from the estimate of the complex OTF ratio; and
(e) repeating steps (a) to (d) using corresponding patches in at least one other part of the common part of the scene represented by the first and second images; and

assembling the determined depth measurements into a depth map of the common part of the scene.

11. A computer readable non-transitory storage medium having a program recorded thereon, the program being executable by a computerized apparatus to determine a depth measurement associated with a scene captured by an image capture device, the program comprising:
code for receiving a first image and a second image of the scene captured by the image capture device, the first image being captured using at least one different camera parameter than that of the second image;
code for selecting a first patch from the first image;
code for selecting a second patch corresponding to the first patch from the second image such that the first and second patches represent a common part of the scene;
code for determining which of the first and second patches is a more focused patch;
code for calculating Fourier transforms for each of the first and second patches;
code for calculating a ratio of the Fourier transforms corresponding to the first and second patches to produce an estimate of a complex OTF ratio, with the Fourier transform of the more focused patch as denominator of the ratio, the estimate of the complex OTF ratio representing relative blur of the first and second patches and being a function of distance to the objects in the common part of the scene; and
code for determining the depth measurement of the common part of the scene from the estimate of the complex OTF ratio.

12. An apparatus for determining a depth measurement associated with a scene captured by an image capture device, the apparatus comprising:
means for receiving a first image and a second image of the scene captured by the image capture device, the first image being captured using at least one different capture parameter than that of the second image;
means for selecting a first patch from the first image;
means for selecting a second patch corresponding to the first patch from the second image such that the first and second patches represent a common part of the scene;
means for determining which of the first and second patches is a more focused patch;
means for calculating Fourier transforms for each of the first and second patches;
means for calculating a ratio of the Fourier transforms corresponding to the first and second patches to produce an estimate of a complex OTF ratio, with the Fourier transform of the more focused patch as denominator of the ratio, the estimate of the complex OTF ratio representing relative blur of the first and second patches and being a function of distance to the objects in the common part of the scene;
means for determining the depth measurement of the common part of the scene from the estimate of the complex OTF ratio.

13. An image capture device, comprising:
a capture system for receiving a first image and a second image of the scene captured by the image capture device, the first image being captured using at least one different capture parameter than that of the second image;
a memory into which the first image and the second image are stored;
a processor configured for determining a depth measurement associated with a scene captured by the first image and the second image, the processor being configured to:
sel

Assignees

• Morgan-Mar David
• Pham Tuan Quang
• Arnison Matthew R
• Larkin Kieran Gerard
• Canon Kk

Inventors

• Morgan-Mar David
• Pham Tuan Quang
• Arnison Matthew R
• Larkin Kieran Gerard

Classifications

• G06T7/571 Primary

  from focus · CPC title

• G06T2207/10148

  Varying focus · CPC title