Method, apparatus and computer program product for image-driven cost volume aggregation

Patent metadata

Field	Value
Publication number	US-9892522-B2
Application number	US-201515116819-A
Country	US
Kind code	B2
Filing date	Feb 9, 2015
Priority date	Feb 14, 2014
Publication date	Feb 13, 2018
Grant date	Feb 13, 2018

Abstract

In an example embodiment, a method, apparatus and computer program product are provided. The method includes computing a cost volume associated with a reference image. Down-sampling of the cost volume and the reference image into at least one level is performed to generate at least one down-sampled cost volume and at least one down-sampled reference image, respectively. An up-sampling of the at least one down-sampled cost volume and the at least one down-sampled reference image into the at least one level is performed to generate at least one up-sampled cost volume and at least one up-sampled reference image, respectively. A color weight map associated with the cost volume and the at least one down-sampled cost volume is computed based on the reference image and the at least one down-sampled reference image at the at least one level. Aggregated cost volume is determined based at least on the color weight map.

First claim

We claim:

1. A method comprising:
computing a cost volume associated with a reference image;
performing down-sampling of the cost volume into at least one level to generate at least one down-sampled cost volume;
performing down-sampling of the reference image into the at least one level to generate at least one down-sampled reference image associated with corresponding at least one down-sampled cost volume;
performing backward up-sampling of the at least one down-sampled cost volume into the at least one level to generate at least one backward up-sampled cost volume;
performing backward up-sampling of the at least one down-sampled reference image associated with the at least one down-sampled cost volume into the at least one level to generate at least one backward up-sampled reference image associated with corresponding backward up-sampled cost volume;
computing at least one color weight map associated with at least one of the cost volume and the at least one down-sampled cost volume based on an associated reference image and the at least one backward up-sampled reference image at the at least one level, the associated reference image being one of the reference image and the at least one down-sampled reference image at the at least one level; and
determining an aggregated cost volume at the at least one level based on a weighted averaging of the at least one backward up-sampled cost volume and an associated cost volume at the at least one level, the associated cost volume being one of the cost volume and the at least one down-sampled cost volume at the at least one level, the weighted averaging being performed based on the at least one color weight map.

2. The method as claimed in claim 1 , wherein the backward up-sampling of the at least one down-sampled cost volume is performed based on respective color parameter value of neighboring pixels associated with an individual pixel in the at least one down-sampled cost volume.

3. The method as claimed in claim 1 , wherein computing the at least one color weight map associated with the cost volume comprises computing a difference between the reference image and a backward up-sampled reference image associated with the reference image.

4. The method as claimed in claim 1 , wherein computing the at least one color weight map associated with the at least one down-sampled cost volume comprises computing a difference between the at least one down-sampled reference image and the at least one backward up-sampled reference image at the at least one level.

5. The method as claimed in claim 4 , wherein the aggregated cost volume is determined based on the following expression:
Ĉ i ( x,y,d )= W i ( x,y )· {tilde over (C)} i ( x,y,d )+(1− W i ( x,y ))· C i ( x,y,d )

where, Ĉ i (x,y,d) is the aggregated cost volume,
W i (x,y) is a color weight map,
C i (x,y,d) is the at least one down-sampled cost volume, and
{tilde over (C)} i (x,y,d) is the at least one backward up-sampled cost volume.

6. The method as claimed in claim 1 , further comprising performing a backward up-sampling of the at least one down-sampled cost volume at a subsequent level based on the aggregated cost volume at the at least one level, the subsequent level being subsequent to the at least one level.

7. The method as claimed in claim 1 , wherein the cost volume is computed based on a matching of a reference image with a plurality of shifted versions of a target image.

8. The method as claimed in claim 7 , wherein the reference image and the target image are rectified images of a scene.

9. The method as claimed in claim 8 , wherein the reference image and the target image comprises a stereoscopic pair of images.

10. An apparatus comprising:
at least one processor; and
at least one memory comprising computer program code, the at least one memory and the computer program code configured to, with the at least one processor, cause the apparatus to at least perform:
compute a cost volume associated with a reference image;
perform down-sampling of the cost volume into at least one level to generate at least one down-sampled cost volume;
perform down-sampling of the reference image into the at least one level to generate at least one down-sampled reference image associated with corresponding at least one down-sampled cost volume;
perform backward up-sampling of the at least one down-sampled cost volume into the at least one level to generate at least one backward up-sampled cost volume;
perform backward up-sampling of the at least one down-sampled reference image associated with the at least one down-sampled cost volume into the at least one level to generate at least one backward up-sampled reference image associated with corresponding backward up-sampled cost volume;
compute at least one color weight map associated with at least one of the cost volume and the at least one down-sampled cost volume based on an associated reference image and the at least one backward up-sampled reference image at the at least one level, the associated reference image being one of the reference image and the at least one down-sampled reference image at the at least one level; and
determine an aggregated cost volume at the at least one level based on a weighted averaging of the at least one backward up-sampled cost volume and an associated cost volume at the at least one level, the associated cost volume being one of the cost volume and the at least one down-sampled cost volume at the at least one level, the weighted averaging being performed based on the at least one color weight map.

11. The apparatus as claimed in claim 10 , wherein the apparatus is further caused, at least in part to perform the backward up-sampling of the at least one down-sampled cost volume based on respective color parameter value of neighboring pixels associated with an individual pixel in the at least one down-sampled cost volume.

12. The apparatus as claimed in claim 10 , wherein for computing the at least one color weight map associated with the cost volume, the apparatus is further caused, at least in part to compute a difference between the reference image and a backward up-sampled reference image associated with the reference image.

13. The apparatus as claimed in claim 10 , wherein for computing the at least one color weight map associated with the at least one down-sampled cost volume, the apparatus is further caused, at least in part to compute a difference between the at least one down-sampled reference image and the at least one backward up-sampled reference image at the at least one level.

14. The apparatus as claimed in claim 12 , wherein the apparatus is further caused, at least in part to determine the aggregated cost volume based on the following expression:
Ĉ i ( x,y,d )= W i ( x,y )· {tilde over (C)} i ( x,y,d )+(1− W i ( x,y ))· C i ( x,y,d )

where, Ĉ i (x,y,d) is the aggregated cost volume,
W i (x,y) is a color weight map,
C i (x,y,d) is the at least one down-sampled cost volume, and
{tilde over (C)} i (x,y,d) is the at least one backward up-sampled cost volume.

15. The apparatus as claimed in claim 10 , wherein the apparatus is further caused, at least in part to perform a backward up-sampling of the a least tone down-sampled cost volume at a subsequent level based on the aggregated cost volume at the at least one level, the subsequent level being subsequent to the at least one level.

16. The apparatus as claimed in claim 10 , wherein the cost volume is computed based on a matching of a reference image with a plurality of shifted versions of a target image.

Assignees

• Nokia Technologies Oy

Inventors

• Smirnov Sergey
• Gotchev Atanas

Classifications

• H04N13/0271

  Electricity · mapped topic

• H04N13/0018

  Electricity · mapped topic

• H04N13/0037

  Electricity · mapped topic

• G06T2207/10024

  Color image · CPC title

• G06T2207/10028

  Range image; Depth image; 3D point clouds · CPC title