Systems and methods for estimating depth and visibility from a reference viewpoint for pixels in a set of images captured from different viewpoints

Patent metadata

Field	Value
Publication number	US-12002233-B2
Application number	US-202117162542-A
Country	US
Kind code	B2
Filing date	Jan 29, 2021
Priority date	Aug 21, 2012
Publication date	Jun 4, 2024
Grant date	Jun 4, 2024

Abstract

Systems in accordance with embodiments of the invention can perform parallax detection and correction in images captured using array cameras. Due to the different viewpoints of the cameras, parallax results in variations in the position of objects within the captured images of the scene. Methods in accordance with embodiments of the invention provide an accurate account of the pixel disparity due to parallax between the different cameras in the array, so that appropriate scene-dependent geometric shifts can be applied to the pixels of the captured images when performing super-resolution processing. In a number of embodiments, generating depth estimates considers the similarity of pixels in multiple spectral channels. In certain embodiments, generating depth estimates involves generating a confidence map indicating the reliability of depth estimates.

First claim

What is claimed is:

1. A method of estimating distances to objects within a scene from a light field comprising a set of images captured from different viewpoints using a processor configured by an image processing application, the method comprising:
selecting a reference image from a reference viewpoint relative to the viewpoints of the set of images captured from different viewpoints;
determining depth estimates for pixel locations in the reference image using at least a subset of the set of images, where a depth estimate for a given pixel location in the reference image is determined by:
dividing the reference image into spatial blocks;
generating depth measurements for a sparser subset of indicator pixels within the spatial blocks, where the spatial block includes a plurality of indicator pixels;
assigning a depth to the pixels in each of the spatial blocks based upon the depths of the indicator pixels.

2. The method of claim 1 , further comprising:
determine a reliability of the depth assigned to each of the spatial blocks;
when a depth transition is detected within spatial blocks, refining the depth of the spatial block by performing additional depth measurements within the spatial bock that includes the depth transitions.

3. The method of claim 1 , wherein a depth of a non-indicator pixels is calculated as a normalized weighted average of the distances to the nearest indicator pixels within a fixed neighborhood.

4. The method of claim 1 , further comprising using nearest neighbor interpolation to assign depths to the pixels in the spatial block based upon the depth measurements of the indicator pixels.

5. The method of claim 1 , wherein assigning a depth to the pixels in each of the spatial blocks is obtained through interpolation of neighboring indicator pixels.

6. The method of claim 5 , wherein weights for the interpolation incorporate intensity similarity and spatial distance to the nearest indicator pixel.

7. The method of claim 1 , further comprising measuring similarity of corresponding pixels using a block-based similarity measure.

8. The method of claim 1 , wherein selecting a reference viewpoint relative to the viewpoints of the set of images captured from different viewpoints comprises selecting a viewpoint from the set consisting of:
the viewpoint of one of the images; and
a virtual viewpoint.

9. The method of claim 1 , further comprising:
identifying pixels as corresponding pixels in the at least a subset of the set of images that correspond to the given pixel location in the reference image based upon expected disparity at a plurality of depths, wherein disparity determines an amount an object shifts between images from different cameras and is based on a baseline distance between the different cameras;
comparing a similarity of intensity values of the corresponding pixels identified in the subset of the set of images at each of the plurality of depths; and
selecting the depth from the plurality of depths at which the identified corresponding pixels have the highest degree of similarity as a depth estimate for the given pixel location in the reference image.

10. The method of claim 9 , wherein comparing a similarity of intensity values of the corresponding pixels comprises using a cost function to determine the similarity of corresponding pixels.

11. The method of claim 9 , wherein identifying corresponding pixels comprises determining that a pixel in a given image from the set of images corresponds to the pixel location in the reference image from the reference viewpoint by applying a scene dependent shift to the pixel location in the reference image from the reference viewpoint that is determined based upon:
a depth estimate of the pixel location in the reference image; and
a baseline between the viewpoint of the given image and the reference viewpoint.

12. The method of claim 9 , further comprising generating confidence metrics for the current depth estimates for pixel locations in the reference image from the reference viewpoint, wherein a confidence metric for a particular pixel location is based on a signal to noise ratio, SNR, in a region of the particular pixel location with which a depth estimate is associated.

13. The method of claim 9 , wherein selecting the depth from the plurality of depths at which the identified corresponding pixels have the highest degree of similarity as the depth estimate for the given pixel location in the reference image from the reference viewpoint further comprises selecting the depth from the plurality of depths at which the cost function for the identified corresponding pixels indicates the highest level of similarity.

14. The method of claim 9 , further comprising fusing pixels from the set of images using the processor configured by the image processing application based upon the depth estimates to create a fused image having a resolution that is greater than the resolutions of the images in the set of images.

15. The method of claim 14 , further comprising synthesizing an image from the reference viewpoint using the processor configured by the image processing application to perform a super-resolution process based upon the fused image from the reference viewpoint, the set of images captured from different viewpoints, and the depth estimates.

16. The method of claim 1 , further comprising:
normalizing the set of images, wherein the normalizing comprises using photometric calibration data to perform photometric normalization.

17. The method of claim 16 , wherein normalizing the set of images further comprises:
utilizing calibration information to correct for photometric variations and scene-independent geometric distortions in the images in the set of images; and
rectification of the images in the set of images.

18. The method of claim 17 , wherein normalizing the set of images further comprises:
resampling the images to increase the similarity of corresponding pixels by correcting for geometric lens distortion; and
applying scene-independent geometric corrections to the images at a sub-pixel resolution.

19. An array camera, comprising:
a plurality of cameras;
a processing system;
memory containing an image processing application;
wherein the processor is configured by the image processing application to estimate distances to objects within a scene by:
selecting a reference image from a reference viewpoint relative to the viewpoints of the set of images captured from different viewpoints;
determining depth estimates for pixel locations in the reference image using at least a subset of the set of images, where a depth estimate for a given pixel location in the reference image is determined by:
dividing the reference image into spatial blocks;
generating depth measurements for a sparser subset of indicator pixels within the spatial blocks, where the spatial block includes a plurality of indicator pixels;
assigning a depth to the pixels in each of the spatial blocks based upon the depths of the indicator pixels.

Assignees

• Adeia Imaging Llc

Inventors

• Ciurea Florian
• Venkataraman Kartik
• Molina Gabriel
• Lelescu Dan

Classifications

• G06T7/593 Primary

  from stereo images · CPC title

• G02B27/0075 Primary

  with means for altering, e.g. increasing, the depth of field or depth of focus · CPC title

• G06T7/557

  from light fields, e.g. from plenoptic cameras · CPC title

• G06T7/85

  Stereo camera calibration · CPC title

• G06T15/20

  Perspective computation · CPC title