Method, apparatus and computer program product for depth estimation

Patent metadata

Field	Value
Publication number	US-9524556-B2
Application number	US-201514705348-A
Country	US
Kind code	B2
Filing date	May 6, 2015
Priority date	May 20, 2014
Publication date	Dec 20, 2016
Grant date	Dec 20, 2016

Abstract

In an example embodiment, a method, apparatus and computer program product are provided. The method includes computing a first cost volume for a light-field image. A first depth map comprising depth information of the plurality of sub-images of the light-field image is computed based on the first cost volume. A first view image comprising reconstruction information is reconstructed based on the depth information of the plurality of sub-images. A second cost volume corresponding to the first cost volume is computed based on the reconstruction information. The second cost volume is filtered based on the first view image to generate an aggregated cost volume. A second depth map is generated based on the aggregated cost volume. The second depth map facilitates generation of a second view image that is associated with a resolution higher than a resolution of the first view image.

First claim

What is claimed is:

1. A method comprising:
computing, for a plurality of disparity values, a first cost volume based on a cost matching between a plurality of pixels associated with a plurality of sub-images of a light-field image;
computing a first depth map based on the first cost volume, the first depth map comprising a depth information of the plurality of sub-images of the light-field image;
reconstructing a first view image associated with the light-field image based on the depth information of the plurality of sub-images, the first view image comprising reconstruction information associated with the plurality of pixels;
computing a second cost volume corresponding to the first cost volume based on the reconstruction information of the first view image;
filtering the second cost volume based on the first view image to generate an aggregated cost volume; and
generating a second depth map based on the aggregated cost volume, the second depth map configured to facilitate in generation of a second view image, wherein resolution of the second view image is higher than a resolution of the first view image.

2. The method as claimed in claim 1 , wherein computing the first cost volume comprises:
computing, for a pixel of the plurality of pixels in a current sub-image, an absolute difference between pixel value of the pixel in the current sub-image and pixel value of a corresponding pixel in a neighboring sub-image; and
generating, for a disparity value of the plurality of disparity values, a slice of a plurality of slices of the first cost volume based on the cost matching.

3. The method as claimed in claim 2 , wherein computing the first depth map comprises:
selecting a size for a support window corresponding to a sub-image of the plurality of sub-images;
computing summed costs by aggregating matching costs of pixels in the support window in the plurality of slices corresponding to the plurality of disparity values; and
assigning, from among the plurality of disparity values, a disparity value corresponding to a minimum summed cost as the disparity value for the sub-image.

4. The method as claimed in claim 3 , wherein reconstructing the first view image comprises:
selecting, from the plurality of sub-images, respective patches based on the disparity value associated with respective sub-images of the plurality of sub-images; and
concatenating the respective patches associated with the plurality of sub-images to reconstruct the first view image.

5. The method as claimed in claim 4 , wherein computing the second cost volume comprises selecting, for the respective patches in the first view image, corresponding patches from the plurality of slices of the first cost volume.

6. The method as claimed in claim 1 , wherein filtering the second cost volume comprises:
performing cost aggregation on the second cost volume based on tree-based aggregation.

7. The method as claimed in claim 1 , further comprising:
comparing a measure of at least one quality parameter associated with the second reference image with a threshold measure; and
performing iteratively, upon determination of the measure of the at least one quality parameter being lower than the threshold measure:
performing cost aggregation on the second cost volume based on the second view image to re-compute the second cost volume,
re-computing the second depth map based on the re-computed second cost volume, and
reconstructing the second reference image based on the recomputed second depth map.

8. The method as claimed in claim 1 , further comprising facilitating receipt of the light-field image of a scene.

9. An apparatus comprising:
at least one processor; and
at least one memory comprising computer program code, the at least one memory and the computer program code configured to, with the at least one processor, cause the apparatus to at least perform:
compute, for a plurality of disparity values, a first cost volume based on a cost matching between plurality of pixels associated with a plurality of sub-images of a light-field image;
compute a first depth map based on the first cost volume, the first depth map comprising a depth information of the plurality of sub-images of the light-field image;
reconstruct a first view image associated with the light-field image based on the depth information of the plurality of sub-images, the first view image comprising reconstruction information associated with the plurality of pixels;
compute a second cost volume corresponding to the first cost volume based on the reconstruction information of the first view image;
filter the second cost volume based on the first view image to generate an aggregated cost volume; and
generate a second depth map based on the aggregated cost volume, the second depth map configured to facilitate in generation of a second view image, wherein resolution of the second view image is higher than a resolution of the first view image.

10. The apparatus as claimed in claim 9 , wherein to compute the first cost volume, the apparatus is caused at least in parts to:
compute, for a pixel of the plurality of pixels in a current sub-image, an absolute difference between pixel value of the pixel in the current sub-image and pixel value of a corresponding pixel in a neighboring sub-image; and
generate, for a disparity value of the plurality of disparity values, a slice of a plurality of slices of the first cost volume based on the cost matching.

11. The apparatus as claimed in claim 10 , wherein to compute the first depth map, the apparatus is further caused, at least in part to:
select a size for a support window corresponding to a sub-image of the plurality of sub-images;
compute summed costs by aggregating matching costs of pixels in the support window in the plurality of slices corresponding to the plurality of disparity values; and
assign, from among the plurality of disparity values, a disparity value corresponding to a minimum summed cost as the disparity value for the sub-image.

12. The apparatus as claimed in claim 11 , wherein to reconstruct the first view image, the apparatus is further caused, at least in part to:
select, from the plurality of sub-images, respective patches based on the disparity value associated with respective sub-images of the plurality of sub-images; and
concatenate the respective patches associated with the plurality of sub-images to reconstruct the first view image.

13. The apparatus as claimed in claim 12 , wherein to compute the second cost volume, the apparatus is further caused, at least in part to select, for the respective patches in the first view image, corresponding patches from the plurality of slices of the first cost volume.

14. The apparatus as claimed in claim 9 , wherein to filter the second cost volume, the apparatus is further caused, at least in part to perform cost aggregation on the second cost volume based on tree-based aggregation.

15. The apparatus as claimed in claim 9 , wherein the apparatus is further caused, at least in part to:
compare a measure of at least one quality parameter associated with the second reference image with a threshold measure; and
perform iteratively, upon determination of the measure of the at least one quality parameter being lower than the threshold measure:
performing cost aggregation on the second cost volume based on the second view image to re-compute the second cost volume,
re-compute the second depth map based on the re-computed second cost volume, and
reconstruct the second reference image based on the recomputed second depth map.

16. The appar

Assignees

• Nokia Corp
• Nokia Technologies Oy

Inventors

• Uliyar Mithun
• Putraya Gururaj Gopal
• Ukil Soumik
• S V Basavaraja
• Govindarao Krishna Annasagar
• Muninder Veldandi

Classifications

• G06T2207/10052

  Images from lightfield camera · CPC title

• G06T2207/20228

  Disparity calculation for image-based rendering · CPC title

• G06T2200/21

  involving computational photography · CPC title

• G06T7/0065

  Physics · mapped topic

• H04N13/0232

  Electricity · mapped topic