Systems and methods for hybrid depth regularization

Patent metadata

Field	Value
Publication number	US-12423849-B2
Application number	US-202418632171-A
Country	US
Kind code	B2
Filing date	Apr 10, 2024
Priority date	Aug 21, 2017
Publication date	Sep 23, 2025
Grant date	Sep 23, 2025

Abstract

Systems and methods for hybrid depth regularization in accordance with various embodiments of the invention are disclosed. In one embodiment of the invention, a depth sensing system comprises a plurality of cameras; a processor; and a memory containing an image processing application. The image processing application may direct the processor to obtain image data for a plurality of images from multiple viewpoints, the image data comprising a reference image and at least one alternate view image; generate a raw depth map using a first depth estimation process, and a confidence map; and generate a regularized depth map. The regularized depth map may be generated by computing a secondary depth map using a second different depth estimation process; and computing a composite depth map by selecting depth estimates from the raw depth map and the secondary depth map based on the confidence map.

First claim

What is claimed is:

1. A depth sensing system, comprising:
a plurality of cameras;
a processor;
a memory containing an image processing application;
wherein the image processing application directs the processor to:
obtain image data for a plurality of images from multiple viewpoints using the plurality of cameras, wherein the image data for the plurality of images comprises a reference image and at least one alternate view image;
identify flat regions in the reference image;
generate initial depth estimates for pixels within the flat regions using the image data for the reference image and the image data for the at least one alternate view image using semi-global matching;
generate a confidence map describing reliability of the generated depth estimates; and
generate a regularized depth map by:
generating secondary depth estimates for pixels using the confidence map; and
computing a composite depth map based upon the initial depth estimates and the secondary depth estimates.

2. The depth sensing system of claim 1 , wherein to identify flat regions, the image processing application further directs the processor to classify pixels as being within the flat regions when a confidence value for a given pixel does not meet a threshold confidence value.

3. The depth sensing system of claim 1 , wherein the image processing application further directs the processor to apply a smoothing filter to the composite depth map.

4. The depth sensing system of claim 1 , wherein the image processing application further directs the processor to:
compute an edge map that indicates pixels within the reference image that form part of an edge; and
compute the composite depth map using the edge map.

5. The depth sensing system of claim 1 , wherein to identify flat regions, the image processing application further directs the processor to:
rectify the reference image and the at least one alternative view image so that rows of pixels in the reference image and the at least one alternative view image correspond to epipolar lines between cameras in the plurality of cameras that captured the reference image and the at least one alternative view image;
determine the number of non-overlapping pixels associated with boundaries by summing non-overlapping pixels in each column of the reference image and the at least one alternative view image; and
identify edges of flat regions based upon columns that include a number of pixels below a threshold number of pixels.

6. The depth sensing system of claim 1 , wherein the image processing application further directs the processor to apply an edge preserving filter to the composite depth map.

7. The depth sensing system of claim 1 , wherein the semi-global matching is performed with downsampling.

8. The depth sensing system of claim 7 , wherein the downsampling is at 1/16 th the resolution of a camera in the plurality of cameras that captured one of the reference image and the at least one alternate view image.

9. The depth sensing system of claim 1 , wherein at least one alternate view image comprises two or more alternate view images, and the image processing application further directs the processor to:
compute a cost volume, for each pair of images containing the reference image and an alternate view image from the two or more alternate view images, with respect to the extent to which pixels from different images at different depths match from the reference image; and
aggregate costs from each cost volume using semi-global matching.

10. The depth sensing system of claim 1 , wherein the image processing application further directs the processor to generate initial depth estimates for pixels outside of the flat regions using calculated parallax between pixels in the reference image and pixels in the at least one alternate view image.

11. A depth sensing method, comprising:
obtaining image data for a plurality of images from multiple viewpoints using the plurality of cameras, wherein the image data for the plurality of images comprises a reference image and at least one alternate view image;
identifying flat regions in the reference image;
generating initial depth estimates for pixels within the flat regions using the image data for the reference image and the image data for the at least one alternate view image using semi-global matching
generating a confidence map describing reliability of the generated depth estimates; and
generating a regularized depth map by:
generating secondary depth estimates for pixels using the confidence map; and
computing a composite depth map based upon the initial depth estimates and the secondary depth estimates.

12. The depth sensing method of claim 11 , wherein the plurality of cameras, the processor, and the memory are part of a mobile phone.

13. The depth sensing method of claim 11 , wherein identifying flat regions comprises classifying pixels as being within the flat regions when a confidence value for a given pixel does not meet a threshold confidence value.

14. The depth sensing method of claim 11 , further comprising applying a smoothing filter to the composite depth map.

15. The depth sensing method of claim 11 , further comprising:
computing an edge map that indicates pixels within the reference image that form part of an edge; and
computing the composite depth map using the edge map.

16. The depth sensing method of claim 11 , wherein to identifying flat regions comprises:
rectifying the reference image and the at least one alternative view image so that rows of pixels in the reference image and the at least one alternative view image correspond to epipolar lines between cameras in the plurality of cameras that captured the reference image and the at least one alternative view image;
determining the number of non-overlapping pixels associated with boundaries by summing non-overlapping pixels in each column of the reference image and the at least one alternative view image; and
identifying edges of flat regions based upon columns that include a number of pixels below a threshold number of pixels.

17. The depth sensing method of claim 11 , further comprising applying an edge preserving filter to the composite depth map.

18. The depth sensing method of claim 11 , further comprising performing the semi-global matching with downsampling.

19. The depth sensing method of claim 18 , wherein the downsampling is at 1/16 th the resolution of a camera in the plurality of cameras that captured one of the reference image and the at least one alternate view image.

20. The depth sensing method of claim 11 , wherein at least one alternate view image comprises two or more alternate view images, the method further comprising:
computing a cost volume, for each pair of images containing the reference image and an alternate view image from the two or more alternate view images, with respect to the extent to which pixels from different images at different depths match from the reference image; and
aggregating costs from each cost volume using semi-global matching.

21. The depth sensing method of claim 11 , further comprising generating initial depth estimates for pixels outside of the flat regions using calculated parallax between pixels in the reference image and pixels in the at least one alternate view image.

22. The depth sensing method of claim 11 , wherein the plurality of cameras, are part of a mobile phone.

Assignees

• Adeia Imaging Llc

Inventors

• Jain Ankit
• Chatterjee Priyam
• Venkataraman Kartik

Classifications

• G06T5/50

  using two or more images, e.g. averaging or subtraction · CPC title

• G06T2207/20032

  Median filtering · CPC title

• G06T2207/20192

  Edge enhancement; Edge preservation · CPC title

• G06T2207/20028

  Bilateral filtering · CPC title

• G06T7/136

  involving thresholding · CPC title