Systems and methods for estimating depth and visibility from a reference viewpoint for pixels in a set of images captured from different viewpoints

Patent metadata

Field	Value
Publication number	US-12437432-B2
Application number	US-202418434684-A
Country	US
Kind code	B2
Filing date	Feb 6, 2024
Priority date	Aug 21, 2012
Publication date	Oct 7, 2025
Grant date	Oct 7, 2025

Abstract

Systems in accordance with embodiments of the invention can perform parallax detection and correction in images captured using array cameras. Due to the different viewpoints of the cameras, parallax results in variations in the position of objects within the captured images of the scene. Methods in accordance with embodiments of the invention provide an accurate account of the pixel disparity due to parallax between the different cameras in the array, so that appropriate scene-dependent geometric shifts can be applied to the pixels of the captured images when performing super-resolution processing. In a number of embodiments, generating depth estimates considers the similarity of pixels in multiple spectral channels. In certain embodiments, generating depth estimates involves generating a confidence map indicating the reliability of depth estimates.

First claim

What is claimed is:

1. An imaging system, comprising:
a plurality of imagers;
a processor; and
a memory containing an image processing application that configures the processing system to:
obtain a plurality of images of a scene comprising a right image and a left image from the plurality of imagers;
rectify the plurality of images;
compute disparities between pixels in the right image and pixels in the left image;
select a reference image from among the right image and the left image;
identify occluded pixels in the reference image; and
generate a confidence map for the computed disparities associated with the reference image, where the confidence map comprises a confidence metric for pixels in the reference image based on the computed disparities, and where the confidence metric for occluded pixels encodes a confidence factor indicating detection of an occlusion.

2. The imaging system of claim 1 , wherein the confidence metric encodes an error.

3. The imaging system of claim 1 , wherein:
the image processing application further configures the processor to:
generate reduced resolution images from the plurality of images; and
computing the disparity between pixels in the right image and pixels in the left image comprises utilizing the reduced resolution images to compute disparity between pixels in the right image and pixels in the left image.

4. The imaging system of claim 1 , wherein the image processing application further configures the processor to obtain calibration information for a left imager and a right imager from the plurality of imagers, where the right image is generated by the right imager, and the left image is generated by the left imager, and the rectification is based on the calibration information.

5. The imaging system of claim 4 , wherein the calibration information comprises a focal length of the right imager and a focal length of the left imager.

6. The imaging system of claim 1 , wherein the image processing application further configures the processor to refine, using the confidence map, the computed disparities to account for occluded pixels.

7. The imaging system of claim 1 , wherein the image processing application further configures the processor to estimate depths of objects within the scene based on the computed disparities and the confidence map.

8. The imaging system of claim 1 , wherein obtaining the plurality of images of the scene comprises capturing a plurality of images using the plurality of imagers; and obtaining a plurality of lower resolution images.

9. The imaging system of claim 1 , wherein to identify occluded pixels, the image processing application further configures the processor to identify likely occlusion zones in the reference image.

10. The imaging system of claim 1 , wherein to identify occluded pixels, the image processing application further configures the processor to:
calculate a photometric distance of corresponding pixels in a non-reference image along an epipolar line with a reference pixel in the reference image; and
determine that a corresponding pixel is occluded when its photometric distance exceeds a threshold.

11. The imaging system of claim 1 , wherein the image processing application further configures the processor to generate a depth map based on the disparities.

12. An imaging method of generating confidence maps for images, comprising:
obtaining a plurality of images of a scene comprising a right image and a left image from a plurality of imagers;
rectifying the plurality of images;
computing disparities between pixels in the right image and pixels in the left image;
selecting a reference image from among the right image and the left image;
identifying occluded pixels in the reference image; and
generating a confidence map for the computed disparities associated with the reference image, where the confidence map comprises a confidence metric for pixels in the reference image based on the computed disparities, and where the confidence metric for occluded pixels encodes a confidence factor indicating detection of an occlusion.

13. The method of claim 12 , wherein the confidence metric encodes an error.

14. The method of claim 12 , further comprising:
generating reduced resolution images from the plurality of images; and
computing the disparity between pixels in the right image and pixels in the left image comprises utilizing the reduced resolution images to compute disparity between pixels in the right image and pixels in the left image.

15. The method of claim 12 , further comprising obtaining calibration information for a left imager and a right imager from the plurality of imagers, where the right image is generated by the right imager, and the left image is generated by the left imager, and the rectification is based on the calibration information.

16. The method of claim 15 , wherein the calibration information comprises a focal length of the right imager and a focal length of the left imager.

17. The method of claim 12 , further comprising refining, using the confidence map, the computed disparities to account for occluded pixels.

18. The method of claim 12 , further comprising estimating depths of objects within the scene based on the computed disparities and the confidence map.

19. The method of claim 12 , wherein obtaining the plurality of images of the scene comprises capturing a plurality of images using the plurality of imagers; and obtaining a plurality of lower resolution images.

20. The method of claim 12 , wherein identifying occluded pixels comprises identifying likely occlusion zones in the reference image.

21. The method of claim 12 , wherein identifying occluded pixels comprises:
calculating a photometric distance of corresponding pixels in a non-reference image along an epipolar line with a reference pixel in the reference image; and
determining that a corresponding pixel is occluded when its photometric distance exceeds a threshold.

22. The method of claim 12 , further comprising generating a depth map based on the disparities.

Assignees

• Adeia Imaging Llc

Inventors

• Ciurea Florian
• Venkataraman Kartik
• Molina Gabriel
• Lelescu Dan

Classifications

• G06T2207/10012

  Stereo images · CPC title

• G06T2207/10024

  Color image · CPC title

• H04N23/16

  Optical arrangements associated therewith, e.g. for beam-splitting or for colour correction · CPC title

• H04N13/232

  using fly-eye lenses, e.g. arrangements of circular lenses · CPC title

• H04N2013/0081

  Depth or disparity estimation from stereoscopic image signals · CPC title