Method, system and apparatus for determining a depth value of a pixel

Patent metadata

Field	Value
Publication number	US-10019810-B2
Application number	US-201414552364-A
Country	US
Kind code	B2
Filing date	Nov 24, 2014
Priority date	Nov 28, 2013
Publication date	Jul 10, 2018
Grant date	Jul 10, 2018

Abstract

A method of determining a depth value of a fine structure pixel in a first image of a scene using a second image of the scene is disclosed. A gradient orientation for each of a plurality of fine structure pixels in the first image is determined. Difference images are generated from the second image and a series of blurred images formed from the first image, each difference image corresponding to one of a plurality of depth values. Each of the difference images is smoothed, in accordance with the determined gradient orientations, to generate smoothed difference images having increased coherency of fine structure. For each of a plurality of fine structure pixels in the first image, one of the smoothed difference images is selected. The depth value of the fine structure pixel corresponding to the selected smoothed difference image is determined.

First claim

The invention claimed is:

1. A method of determining a depth value of a fine structure pixel in a first image of a scene using a second image of the scene, the method comprising:
determining a gradient orientation for each of a plurality of fine structure pixels in the first image;
generating difference images from the second image and a series of blurred images formed from said first image, each said difference image corresponding to one of a plurality of depth values;
generating smoothed difference images by applying oriented smoothing to each of the difference images with an angle of the oriented smoothing being determined according to the determined gradient orientation, the smoothed difference images having increased coherency of fine structure;
selecting, for each of the plurality of fine structure pixels in the first image, one of the generated smoothed difference images having a minimum smoothed blur difference value at a corresponding pixel of the generated smoothed difference image; and
determining the depth value of the fine structure pixel corresponding to the selected smoothed difference image.

2. The method according to claim 1 , wherein the gradient orientations are determined for each of a plurality of pixels in a fine structure region of the first image.

3. The method according to claim 1 , further comprising smoothing pixel values in a region of each of the set of difference images, the difference image regions corresponding to a fine structure region in the first image, so as to generate said smoothed difference images.

4. The method according to claim 1 , wherein said smoothed difference image is selected according to a difference value at a pixel.

5. The method according to claim 1 , further comprising determining a depth measurement for fine structure in the scene from the depth value associated with each of a plurality of pixels in a fine structure region of the first image.

6. The method according to claim 1 , wherein the difference images are smoothed using edge tangent convolution.

7. The method according to claim 1 , wherein the difference images are smoothed using a line integral convolution.

8. The method according to claim 1 , further comprising determining a confidence value for each of said pixels in the first image from the difference between a maximum and a minimum of a subset of difference values of the smoothed difference images at each of said pixels in the first image, wherein said subset of difference values corresponds to smoothed difference images that lie within a predetermined depth of each of said pixels in the first image.

9. The method according to claim 1 , further comprising:
determining a confidence value for each of said pixels in the first image from a fluctuation in difference values of the smoothed difference images at each of said pixels in the first image, wherein said fluctuation is determined from the difference between a minimum difference value and the difference value of the smoothed difference image, and a sum of the absolute difference between difference values of the smoothed difference images of successive depth and local cross-correlation of said first and second captured images.

10. The method according to claim 1 , further comprising blurring the first captured image or the second captured image depending on the relation between the depth and the focus bracket step size.

11. The method according to claim 1 , wherein the difference images are generated based on a direct search over a predetermined number of steps in depth within a working range.

12. The method according to claim 1 , wherein the difference images are generated based on an optimised search for a minimum smoothed blur difference.

13. The method according to claim 1 , wherein the gradient orientations are determined using the smoothed local gradient.

14. The method according to claim 1 , wherein the gradient orientations are determined using the gradient structure tensor.

15. The method according to claim 1 , wherein the first and second image are captured using different camera parameters including at least one of focus and aperture.

16. The method according to claim 1 , further comprising selecting a pixel of the first image based on a threshold on gradient magnitude of the first image.

17. The method according to claim 1 , further comprising selecting a pixel of the first image based on a threshold on local variance of the first image.

18. An apparatus for determining a depth value of a fine structure pixel in a first image of a scene using a second image of the scene, the apparatus comprising:
a memory for storing data and a computer program;
a processor coupled to the memory for executing the computer program, the computer program having instructions for:
determining a gradient orientation for each of a plurality of fine structure pixels in the first image;
generating difference images from the second image and a series of blurred images formed from said first image, each said difference image corresponding to one of a plurality of depth values;
generating smoothed difference images by applying oriented smoothing to each of the difference images with an angle of the oriented smoothing being determined according to the determined gradient orientation, the smoothed difference images having increased coherency of fine structure;
selecting, for each of the plurality of fine structure pixels in the first image, one of the generated smoothed difference images having a minimum smoothed blur difference value at a corresponding pixel of the generated smoothed difference image; and
determining the depth value of the fine structure pixel corresponding to the selected smoothed difference image.

19. A non-transitory computer readable medium comprising a computer program stored thereon for determining a depth value of a fine structure pixel in a first image of a scene using a second image of the scene, the program comprising:
code for determining a gradient orientation for each of a plurality of fine structure pixels in the first image;
code for generating difference images from the second image and a series of blurred images formed from said first image, each said difference image corresponding to one of a plurality of depth values;
code for generating smoothed difference images by applying oriented smoothing to each of the difference images with an angle of the oriented smoothing being determined according to the determined gradient orientation, the smoothed difference images having increased coherency of fine structure;
code for selecting, for each of the plurality of fine structure pixels in the first image, one of the generated difference images having a minimum smoothed blur difference value at a corresponding pixel of the generated smoothed difference image; and
code for determining the depth value of the fine structure pixel corresponding to the selected smoothed difference image.

20. A method of determining a confidence value of a fine structure pixel in a first image of a scene using a second image of the scene, the method comprising:
generating difference images from the second image and a series of blurred images formed from the first image, each said difference image corresponding to one of a plurality of depth values; and
determining the confidence value of the fine structure pixel in the first image based on a difference between a maximum and minimum of a subset of blur difference values at a corresponding pixel of the difference images, wherein the subset of blur diffe

Assignees

• Canon Kk

Inventors

• Arnison Matthew Raphael
• Wan Ernest Yiu Cheong

Classifications

• G06T2207/10148

  Varying focus · CPC title

• G06T2207/20224

  Image subtraction · CPC title

• G06T2207/10016

  Video; Image sequence · CPC title

• G06T7/571 Primary

  from focus · CPC title