Scene reconstruction from high spatio-angular resolution light fields

Patent metadata

Field	Value
Publication number	US-9786062-B2
Application number	US-201313944337-A
Country	US
Kind code	B2
Filing date	Jul 17, 2013
Priority date	May 6, 2013
Publication date	Oct 10, 2017
Grant date	Oct 10, 2017

Abstract

The disclosure provides an approach for estimating depth in a scene. According to one aspect, regions where the depth estimation is expected to perform well may first be identified in full-resolution epipolar-plane images (EPIs) generated from a plurality of images of the scene. Depth estimates for EPI-pixels with high edge confidence are determined by testing a number of discrete depth hypotheses and picking depths that lead to highest color density of sampled EPI-pixels. The depth estimate may also be propagated throughout the EPIs. This process of depth estimation and propagation may be iterated until all EPI-pixels with high edge confidence have been processed, and all EPIs may also be processed in this manner. The EPIs are then iteratively downsampled to coarser resolutions, at which edge confidence for EPI-pixels not yet processed are determined, depth estimates of EPI-pixels with high edge confidence made, and depth estimates propagated throughout the EPIs.

First claim

What is claimed is:

1. A computer-implemented method, comprising:
receiving a plurality of images of a scene; and
estimating, from the received images, depths of scene points at multiple image resolution levels beginning from a finest resolution level and iteratively proceeding to coarser resolution levels,
wherein, during each iteration, a respective resolution level is processed by performing steps including:
determining, based on at least local color variation, edge confidence scores for scene points for which depth estimates have not been assigned at any previous iteration,
determining sufficiently detailed regions of the scene at the resolution level, the sufficiently detailed regions being regions in which the edge confidence scores of scene points exceed a threshold value, and
determining depth estimates for scene points in the sufficiently detailed regions.

2. The method of claim 1 , wherein:
estimating depths of scene points at the multiple image resolution levels includes generating epipolar-plane images (EPIs) from the plurality of images, the EPIs being iteratively downsampled to coarser resolutions; and
during each of the iterations, for each of the EPIs:
determining the edge confidence scores for the scene points includes determining edge confidence scores for pixels of the EPI,
determining depth estimates for scene points in the sufficiently detailed regions includes determining depth estimates for pixels of the EPI associated with edge confidence scores above the threshold value, and
the determined depth estimates are propagated to other pixels of the EPI.

3. The method of claim 2 , wherein the plurality of images have optical centers distributed along a one-dimensional (1D) line, capturing a three-dimensional (3D) light field.

4. The method of claim 3 , wherein the depth estimates are determined per scanline.

5. The method of claim 2 , further comprising, during each of the iterations:
determining scores indicating reliability of the depth estimates made during the iteration; and
discarding depth estimates made during the iteration that are associated with reliability scores less than a reliability threshold value.

6. The method of claim 2 , wherein determining a depth estimate for one of the pixels of the EPI includes:
sampling, based on each of a plurality of hypothetical depths, radiances of pixels in the EPI corresponding to respective hypothetical depths; and
assigning depth of the one of the pixels as one of the hypothetical depths that is associated with sampled radiances most densely positioned in an underlying color space.

7. The method of claim 2 , wherein, in determining depth estimates for the pixels of the EPI during each of the iterations, one or more depth estimates determined at previous iterations, if any, are used as depth bounds.

8. The method of claim 2 , further comprising, applying a median filter on the depth estimates.

9. The method of claim 8 , wherein the median filter is a bilateral median filter which is aware of local radiance distribution, edge confidence, and availability of depth estimation.

10. The method of claim 1 , wherein determining a depth estimate for one of the scene points in the sufficiently detailed regions includes:
sampling, based on each of a plurality of hypothetical depths, radiances of pixels, corresponding to the respective hypothetical depths, in a light field constructed from the received images, wherein poses of a camera are estimated to determine sampled rays for the hypothetical depths; and
assigning depth of the one of the scene points as one of the hypothetical depths that is associated with sampled radiances most densely positioned in an underlying color space.

11. The method of claim 1 , wherein the plurality of images have optical centers horizontally and vertically displaced from each other, capturing a four-dimensional (4D) light field.

12. The method of claim 1 , wherein the plurality of images are captured in an unstructured manner.

13. The method of claim 1 , wherein the plurality of images are captured by one of a camera array and a moving camera.

14. A non-transitory computer-readable storage medium storing instructions, which when executed by a computer system, perform operations comprising:
receiving a plurality of images of a scene; and
estimating, from the received images, depths of scene points at multiple resolution levels beginning from a finest resolution level and iteratively proceeding to coarser resolution levels,
wherein, during each iteration, a respective resolution level is processed by performing steps including:
determining, based on at least local color variation, edge confidence scores for scene points for which depth estimates have not been assigned at any previous iteration,
determining sufficiently detailed regions of the scene at the resolution level, the sufficiently detailed regions being regions in which the edge confidence scores of scene points exceed a threshold value, and
determining depth estimates for scene points in the sufficiently detailed regions.

15. The computer-readable storage medium of claim 14 , wherein:
estimating depths of scene points at the multiple image resolution levels includes generating epipolar-plane images (EPIs) from the plurality of images, the EPIs being iteratively downsampled to coarser resolutions; and
during each of the iterations, for each of the EPIs:
determining the edge confidence scores for the scene points includes determining edge confidence scores for pixels of the EPI,
determining depth estimates for scene points in the sufficiently detailed regions includes determining depth estimates for pixels of the EPI associated with edge confidence scores above the threshold value, and
the determined depth estimates are propagated to other pixels of the EPI.

16. The computer-readable storage medium of claim 15 , wherein the plurality of images have optical centers distributed along a one-dimensional (1D) line, capturing a three-dimensional (3D) light field.

17. The computer-readable storage medium of claim 16 , wherein the depth estimates are determined per scanline.

18. The computer-readable storage medium of claim 15 , the operations further comprising, during each of the iterations:
determining scores indicating reliability of the depth estimates made during the iteration; and
discarding depth estimates made during the iteration that are associated with reliability scores less than a reliability threshold value.

19. The computer-readable storage medium of claim 15 , wherein determining a depth estimate for one of the pixels of the EPI includes:
sampling, based on each of a plurality of hypothetical depths, radiances of pixels in the EPI corresponding to respective hypothetical depths; and
assigning depth of the one of the pixels as one of the hypothetical depths that is associated with sampled radiances most densely positioned in an underlying color space.

20. The computer-readable storage medium of claim 15 , wherein, in determining depth estimates for the pixels of the EPI during each of the iterations, one or more depth estimates determined at previous iterations, if any, are used as depth bounds.

21. The computer-readable storage medium of claim 15 , the operations further comprising, applying a median filter on the depth estimates.

22. The computer-readable storage medium of claim 21 , wherein the median filter is a bilateral median filter which is aware of local radiance distr

Assignees

• Disney Entpr Inc
• Eth Zurich (Eidgenoessische Technische Hochschule Zurich)

Inventors

• Sorkine-Hornung Alexander
• Kim Changil
• Zimmer Henning
• Pritch Yael
• Gross Markus

Classifications

• G06T2207/10016

  Video; Image sequence · CPC title

• G06T2207/10052

  Images from lightfield camera · CPC title

• G06T7/557 Primary

  from light fields, e.g. from plenoptic cameras · CPC title

• G06T7/50

  Depth or shape recovery · CPC title

• G06T7/593

  from stereo images · CPC title