Method for alignment of low-quality noisy depth map to the high-resolution colour image

Patent metadata

Field	Value
Publication number	US-10453249-B2
Application number	US-201515521883-A
Country	US
Kind code	B2
Filing date	Oct 23, 2015
Priority date	Oct 31, 2014
Publication date	Oct 22, 2019
Grant date	Oct 22, 2019

Abstract

Various embodiments are provided which relate to the field of image signal processing, specifically relating to the generation of a depth-view image of a scene from a set of input images of a scene taken at different cameras of a multi-view imaging system. A method comprises obtaining a frame of an image of a scene and a frame of a depth map regarding the frame of the image. A minimum depth and a maximum depth of the scene and a number of depth layers for the depth map are determined. Pixels of the image are projected to the depth layers to obtain projected pixels on the depth layers; and cost values for the projected pixels are determined. The cost values are filtered and a filtered cost value is selected from a layer to obtain a depth value of a pixel of an estimated depth map.

First claim

The invention claimed is:

1. A method comprising:
obtaining a frame of an image of a scene;
obtaining a frame of a depth map regarding the frame of the image;
determining a minimum depth and a maximum depth of the scene;
determining a number of depth layers for the depth map;
with at least a processor, projecting pixels of the image to the depth layers to obtain respective projected pixels on the depth layers;
determining cost values for the respective projected pixels;
filtering the cost values;
selecting a filtered cost value from a layer to obtain respective depth values of the pixels of an estimated depth map; and
generating a depth-view image of the scene based on the respective depth values of the pixels.

2. The method according to claim 1 further comprising quantizing the respective depth values.

3. The method according to claim 1 further comprising interpolating the respective depth values.

4. The method according to claim 1 further comprising upsampling the respective depth values to correspond with a resolution of the image.

5. The method according to claim 1 , wherein the determining the cost values comprises using confidence information of at least one of the depth values to replace a low confidence depth value with a predetermined value.

6. The method according to claim 1 , wherein the filtering the cost values comprises using a colour-weighted filter.

7. The method according to claim 1 , wherein the selecting comprises selecting a minimum cost value among cost values of different layers of a current pixel as the depth value for the current pixel.

8. An apparatus configured to:
obtain a frame of an image of a scene;
obtain a frame of a depth map regarding the frame of the image;
determine a minimum depth and a maximum depth of the scene;
determine a number of depth layers for the depth map;
project pixels of the image to the depth layers to obtain respective projected pixels on the depth layers;
determine cost values for the respective projected pixels;
filter the cost values;
select a filtered cost value from a layer to obtain respective depth values of the pixels of an estimated depth map; and
generate a depth-view image of the scene based on the respective depth values of the pixels.

9. The apparatus according to claim 8 , wherein the apparatus is further configured to quantize the respective depth values.

10. The apparatus according to claim 8 , wherein the apparatus is further configured to interpolate the respective depth values.

11. The apparatus according to claim 8 , wherein the apparatus is further configured to upsample the respective depth values to correspond with a resolution of the image.

12. The apparatus according to claim 8 , wherein to determine the cost value, the apparatus is further configured to use confidence information of at least one of the depth values to replace a low confidence depth value with a predetermined value.

13. The apparatus according to claim 8 , wherein to filter the cost vales, the apparatus is further configured to use a colour-weighted filter.

14. The apparatus according to claim 8 , wherein to select the filtered cost value, the apparatus is further configured to select a minimum cost value among cost values of different layers of a current pixel as the depth value for the current pixel.

15. A computer program product comprising computer instructions residing in a non-transitory computer-readable medium, the instructions when executed by a processor cause an apparatus to:
obtain a frame of an image of a scene;
obtain a frame of a depth map regarding the frame of the image;
determine a minimum depth and a maximum depth of the scene;
determine a number of depth layers for the depth map;
project pixels of the image to the depth layers to obtain respective projected pixels on the depth layers;
determine cost values for the respective projected pixels;
filter the cost values;
select a filtered cost value from a layer to obtain respective depth values of the pixels of an estimated depth map; and
generate a depth-view image of the scene based on the respective depth values of the pixels.

16. The computer program product according to claim 15 , wherein the apparatus is further caused to quantize the respective depth values.

17. The computer program product according to claim 15 , wherein the apparatus is further caused to interpolate the respective depth values.

18. The computer program product according to claim 15 , wherein the apparatus is further caused to upsample the respective depth values to correspond with a resolution of the image.

19. The computer program product according to claim 15 , wherein to determine the cost values, the apparatus is further caused to use confidence information of at least one of the depth value to replace a low confidence depth value with a predetermined value.

20. The computer program product according to 15 , wherein to filter the cost values, the apparatus is further caused to use a colour-weighted filter.

21. The computer program product according to claim 15 , wherein to select the filtered cost value, the apparatus is further caused to select a minimum cost value among cost values of different layers of a current pixel as the depth value for the current pixel.

Assignees

• Nokia Technologies Oy

Inventors

• Smirnov Sergey
• Georgiev Mihail
• Gotchev Atanas

Classifications

• H04N13/111

  Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation · CPC title

• H04N13/282

  for generating image signals corresponding to three or more geometrical viewpoints, e.g. multi-view systems · CPC title

• G06T3/4007

  based on interpolation, e.g. bilinear interpolation (image demosaicing G06T3/4015; edge-driven or edge-based scaling G06T3/403) · CPC title

• G06T2207/10028

  Range image; Depth image; 3D point clouds · CPC title

• H04N2213/003

  Aspects relating to the "2D+depth" image format · CPC title