Systems and methods for hybrid depth regularization

Patent metadata

Field	Value
Publication number	US-10482618-B2
Application number	US-201715682295-A
Country	US
Kind code	B2
Filing date	Aug 21, 2017
Priority date	Aug 21, 2017
Publication date	Nov 19, 2019
Grant date	Nov 19, 2019

Abstract

Systems and methods for hybrid depth regularization in accordance with various embodiments of the invention are disclosed. In one embodiment of the invention, a depth sensing system comprises a plurality of cameras; a processor; and a memory containing an image processing application. The image processing application may direct the processor to obtain image data for a plurality of images from multiple viewpoints, the image data comprising a reference image and at least one alternate view image; generate a raw depth map using a first depth estimation process, and a confidence map; and generate a regularized depth map. The regularized depth map may be generated by computing a secondary depth map using a second different depth estimation process; and computing a composite depth map by selecting depth estimates from the raw depth map and the secondary depth map based on the confidence map.

First claim

What is claimed is:

1. A depth sensing system, comprising:
a plurality of cameras;
a processor;
a memory containing an image processing application;
wherein the image processing application directs the processor to:
obtain image data for a plurality of images from multiple viewpoints using the plurality of cameras, wherein the image data for the plurality of images comprises a reference image and at least one alternate view image;
generate a raw depth map containing depth estimates for pixels within the reference image using the image data for the reference image and the image data for the at least one alternate view image using a first depth estimation process, and a confidence map describing reliability of depth estimates contained within the raw depth map; and
generate a regularized depth map by:
computing a secondary depth map containing depth estimates for pixels within the reference image using a second depth estimation process, wherein the second depth estimation process calculates depth using the image data for the reference image and the image data for the at least one alternate view image, by using a different estimation technique than that of the first depth estimation process; and
computing a composite depth map by selecting depth estimates from the raw depth map and the secondary depth map, where a depth estimate for a pixel in the reference image is selected from the raw depth map when the depth estimate is indicated as being reliable by the confidence map.

2. The depth sensing system of claim 1 , wherein the composite depth map is a first composite depth map and the image processing application further directs the processor to generate a regularized depth map by:
computing a second composite depth map by:
applying a smoothing filter to depth estimates in the first composite depth map;
computing an edge map using the filtered first composite depth map, where the edge map indicates pixels within the reference image that form part of an edge; and
computing a second composite depth map by selecting depth estimates from the raw depth map and the filtered first composite depth map, where a depth estimate for a pixel in the reference image is selected from the raw depth map when the edge map indicates that the depth estimate is with respect to a pixel from the reference image that forms part of an edge.

3. The depth sensing system of claim 2 , wherein the image processing application further directs the processor to apply an edge preserving filter to at least one region of the second composite depth map indicated as containing a depth discontinuity by the edge map.

4. The depth sensing system of claim 2 , wherein the image processing application further directs the processor to compute the second composite depth map by:
applying edge detection to the filtered first composite depth map to result in an edge-detected depth map; and
dilating the edge-detected depth map to result in the edge map.

5. The depth sensing system of claim 1 , wherein the image processing application further directs the processor to generate a raw depth map containing depth estimates for pixels within the reference image using the image data for the reference image and the image data for the at least one alternate view image, and a confidence map describing reliability of depth estimates contained within the depth map, by measuring parallax observable between the reference image and the at least one alternate view image.

6. The depth sensing system of claim 5 , wherein the image processing application further directs the processor to estimate depth based upon measured parallax observable between the reference image and the at least one alternate view image by:
measuring parallax by comparing the similarity of a pixel in the reference image to pixels in the at least one alternate view image determined based upon a plurality of depth samples using a cost function; and
estimating depth for the pixel in the reference image by identifying the sampled depth at which the cost function for a pixel in the reference image indicates the strongest match as being the estimated depth of the pixel.

7. The depth sensing system of claim 6 , wherein the image processing application further directs the processor to:
construct a cost volume in memory using costs determined using the cost function at each sampled depth; and
compute the secondary depth map based on the cost volume.

8. The depth sensing system of claim 7 , wherein the image processing application further directs the processor to compute the secondary depth map using a second depth estimation process comprising:
downsampling the reference image and the cost volume; and
estimating depths to enforce smooth depth transitions within the secondary depth map in textureless regions of the reference image using the cost volume.

9. The depth sensing system of claim 1 , wherein the image processing application further directs the processor to compute the composite depth map by determining that a depth estimate for a pixel from the raw depth map is reliable when a confidence value for the pixel from the confidence map is above a threshold.

10. The depth sensing system of claim 1 , wherein the image processing application further directs the processor to:
identify at least one corresponding region within the reference image and the at least one alternate view image using at least one criterion;
generate binary images with respect to the reference image and the at least one alternate view image based upon the identified at least one corresponding region;
subtract shifted versions of the reference image and the at least one alternate view image when the shifts correspond to a plurality of depth samples to produce a shifted and subtracted image for each of the plurality of depth samples;
identify boundaries within the shifted and subtracted images;
identify areas of the boundaries within the shifted and subtracted images; and
determine depth estimates for pixels at a boundary of an identified region in the reference image based upon the depth sample at which the area of the boundary in the shifted and subtracted image is minimized.

11. A depth sensing method, comprising:
obtaining image data for a plurality of images from multiple viewpoints using the plurality of cameras, wherein the image data for the plurality of images comprises a reference image and at least one alternate view image;
generating a raw depth map containing depth estimates for pixels within the reference image using the image data for the reference image and the image data for the at least one alternate view image using a first depth estimation process, and a confidence map describing reliability of depth estimates contained within the raw depth map; and
generating a regularized depth map by:
computing a secondary depth map containing depth estimates for pixels within the reference image using a second depth estimation process, wherein the second depth estimation process calculates depth using the image data for the reference image and the image data for the at least one alternate view image, by using a different estimation technique than that of the first depth estimation process; and
computing a composite depth map by selecting depth estimates from the raw depth map and the secondary depth map, where a depth estimate for a pixel in the reference image is selected from the raw depth map when the depth estimate is indicated as being reliable by the confidence map.

12. The depth sensing method of claim 11 , wherein the composite depth map is a first composite depth map and generating a regularized depth map further comprises:
computing a second composite dep

Assignees

• Fotonation Ltd

Inventors

• Jain Ankit
• Chatterjee Priyam
• Venkataraman Kartik

Classifications

• G06T2207/20028

  Bilateral filtering · CPC title

• G06T2207/20192

  Edge enhancement; Edge preservation · CPC title

• G06T5/30 Primary

  Erosion or dilatation, e.g. thinning · CPC title

• G06T2207/20036

  Morphological image processing · CPC title

• G06T2207/20032

  Median filtering · CPC title