Systems and methods for estimating depth and visibility from a reference viewpoint for pixels in a set of images captured from different viewpoints

Patent metadata

Field	Value
Publication number	US-10380752-B2
Application number	US-201715858974-A
Country	US
Kind code	B2
Filing date	Dec 29, 2017
Priority date	Aug 21, 2012
Publication date	Aug 13, 2019
Grant date	Aug 13, 2019

Abstract

Systems in accordance with embodiments of the invention can perform parallax detection and correction in images captured using array cameras. Due to the different viewpoints of the cameras, parallax results in variations in the position of objects within the captured images of the scene. Methods in accordance with embodiments of the invention provide an accurate account of the pixel disparity due to parallax between the different cameras in the array, so that appropriate scene-dependent geometric shifts can be applied to the pixels of the captured images when performing super-resolution processing. In a number of embodiments, generating depth estimates considers the similarity of pixels in multiple spectral channels. In certain embodiments, generating depth estimates involves generating a confidence map indicating the reliability of depth estimates.

First claim

What is claimed is:

1. An array camera, comprising:
a plurality of cameras;
a processor;
memory containing an image processing application and calibration data;
wherein the processor is configured by the image processing application to produce an image by:
directing the plurality of cameras to capture a plurality of images including a reference image and at least one alternate view image;
retrieving the calibration data from memory;
rectifying the plurality of captured images using the calibration data;
performing a calibration process;
determining a range of disparities to search based upon characteristics of at least one of the plurality of images;
measuring similarity of corresponding pixels in the plurality of images at a plurality of disparities;
generating an initial depth map; and
refocusing the reference image using the initial depth map;
wherein the processor is configured by the image processing application to measure similarity of corresponding pixels in the plurality of images at the plurality of disparities using a cost function.

2. The array camera of claim 1 , wherein the calibration process comprises: obtaining additional scene-independent geometric corrections that compensate for scene-independent geometric distortions that remain after rectification using offline the calibration data.

3. The array camera of claim 2 , wherein scene-independent geometric corrections determined using rectification are determined at a sub-pixel resolution.

4. The array camera of claim 1 , wherein the processor is configured by the image processing application to:
determine characteristics of at least one of the plurality of images;
determine the range of disparities to search based upon the determined characteristics.

5. The array camera of claim 1 , wherein the processor is configured by the image processing application to select a set of disparities to search across the range of disparities.

6. The array camera of claim 5 , wherein the set of disparities do not provide uniform sampling of disparity.

7. The array camera of claim 1 , wherein the processor is configured by the image processing application to select a set of disparities to search across the range of disparities based upon prior information about where objects are in the at least one of the plurality of images.

8. The array camera of claim 1 , wherein the processor is configured by the image processing application to measure similarity of corresponding pixels using a block-based similarity measure.

9. The array camera of claim 1 , wherein the processor is configured by the image processing application to filter calculated costs using an edge preserving filter.

10. The array camera of claim 9 , wherein the processor is configured by the image processing application to filter the calculated costs using a bilateral filter.

11. The array camera of claim 10 , wherein the processor is configured by the image processing application to determine filter weights of the bilateral filter based upon the reference image and the resulting filter weights are applied to the calculated costs.

12. The array camera of claim 1 , wherein the processor is configured by the image processing application to identify pixels that may be occluded using the initial depth map by identifying pixel depth values that are inconsistent with local pixel depth values.

13. The array camera of claim 1 , wherein the processor is configured by the image processing application to identify occluded pixels based upon scene-dependent geometric shifts of the pixels.

14. The array camera of claim 1 , wherein the processor is configured by the image processing application to generate a depth of an occluded pixel by averaging depth of surrounding pixels.

15. The array camera of claim 1 , wherein the processor is configured by the image processing application to generate a depth of an occluded pixel by interpolating depths of adjacent pixels for which depths have been calculated.

16. The array camera of claim 1 , wherein the processor is configured by the image processing application to generate a depth estimate for a given pixel location in the initial depth map by:
identifying at least one pixel in the plurality of images that corresponds to the given pixel location in the reference image based upon expected disparity at the plurality of depths;
comparing the similarity of the pixels that are identified as corresponding at each of the plurality of depths; and
selecting a depth from the plurality of depths at which the identified corresponding pixels have a highest degree of similarity as a depth estimate for the given pixel location.

17. The array camera of claim 16 , wherein the processor is configured by the image processing application to select the depth from the plurality of depths at which the identified corresponding pixels have the highest degree of similarity as the depth estimate for the given pixel location by selecting the depth from the plurality of depths at which a spatially filtered cost function for the identified corresponding pixels indicates the highest level of similarity.

18. The array camera of claim 16 , wherein the processor is configured by the image processing application to compare the similarity of the pixels that are identified as corresponding at each of the plurality of depths by comparing intensity values of corresponding pixels across color channels.

Assignees

• Fotonation Ltd

Inventors

• Ciurea Florian
• Venkataraman Kartik
• Molina Gabriel
• Lelescu Dan

Classifications

• H04N23/16

  Optical arrangements associated therewith, e.g. for beam-splitting or for colour correction · CPC title

• H04N2013/0088

  Synthesising a monoscopic image signal from stereoscopic images, e.g. synthesising a panoramic or high resolution monoscopic image · CPC title

• G06T2200/21

  involving computational photography · CPC title

• G06T7/85

  Stereo camera calibration · CPC title

• H04N13/243

  using three or more two-dimensional [2D] image sensors · CPC title