Method, system and apparatus for forming a high resolution depth map

Patent metadata

Field	Value
Publication number	US-2016063715-A1
Application number	US-201514835475-A
Country	US
Kind code	A1
Filing date	Aug 25, 2015
Priority date	Aug 27, 2014
Publication date	Mar 3, 2016
Grant date	—

Abstract

An input depth map is received having less resolution than an image. From a window region about an image pixel of the image, pixels are selected having a substantially similar characteristic value as the image pixel. A reference depth value is determined for the image pixel from depth values in the input depth map which correspond to each of the selected pixels. Weights are determined for each pixel within the window region based on a difference between a depth value corresponding to each of the selected pixels and the determined reference depth value for the image pixel. A refined depth value is determined for the image pixel from a weighted sum of the depth values corresponding to each of the selected pixels. A high resolution depth map corresponding to the image is determined from the refined depth value.

First claim

1 . A method of determining a high resolution depth map corresponding to an image of a scene, said method comprising:
receiving the image and an input depth map corresponding to the image; selecting, from a window region about an image pixel of the image, a plurality of pixels having a substantially similar characteristic value as the image pixel; determining a reference depth value for the image pixel from depth values in the input depth map which correspond to each of the selected pixels; determining weights for each pixel within the window region based at least on a difference between a depth value corresponding to each of the selected pixels and the determined reference depth value for the image pixel; determining a refined depth value for the image pixel from a weighted sum of the depth values corresponding to each of the selected pixels, said depth values being weighted in accordance with the determined weights; and determining a high resolution depth map corresponding to the image from the refined depth value.

2 . The method according to claim 1 , wherein a window region is determined for each image pixel of said image.

3 . The method according to claim 1 , further comprising determining an image similarity measure between the image pixel and at least one of said selected pixels.

4 . The method according to claim 1 , further comprising determining an image similarity measure between the image pixel and at least one of said selected pixels, wherein the reference depth value is based on median depth of said selected pixels.

5 . The method according to claim 1 , further comprising determining an image similarity measure between the image pixel and at least one of said selected pixels, wherein the reference depth value is based on the image similarity measure.

6 . The method according to claim 1 , wherein the high resolution depth map is determined using an occlusion mask corresponding to image.

7 . The method according to claim 1 , further comprising selecting an occluded region within the input depth map wherein the high resolution depth map is determined using an occlusion mask corresponding to image.

8 . The method according to claim 1 , further comprising selecting an occluded region within the input depth map, wherein the high resolution depth map is determined using an occlusion mask corresponding to image, and wherein depth values are interpolated with the occluded region.

9 . The method according to claim 1 , further comprising interpolating depth values for the image pixel.

10 . The method according to claim 1 , further comprising interpolating depth values for the image pixel, wherein the depth values for the image pixel are interpolated as a weighted sum of depth values corresponding to each of the selected pixels.

11 . A system for determining a high resolution depth map corresponding to an image of a scene, said system comprising:
a memory for storing data and a computer program; a processor coupled to the memory for executing the computer program, the computer program comprising instructions for:
receiving the image and an input depth map corresponding to the image;
selecting, from a window region about an image pixel of the image, a plurality of pixels having a substantially similar characteristic value as the image pixel;
determining a reference depth value for the image pixel from depth values in the input depth map which correspond to each of the selected pixels;
determining weights for each pixel within the window region based at least on a difference between a depth value corresponding to each of the selected pixels and the determined reference depth value for the image pixel;
determining a refined depth value for the image pixel from a weighted sum of the depth values corresponding to each of the selected pixels, said depth values being weighted in accordance with the determined weights; and
determining a high resolution depth map corresponding to the image from the refined depth value.

12 . An apparatus for determining a high resolution depth map corresponding to an image of a scene, said apparatus comprising:
means for receiving the image and an input depth map corresponding to the image; means for selecting, from a window region about an image pixel of the image, a plurality of pixels having a substantially similar characteristic value as the image pixel; means for determining a reference depth value for the image pixel from depth values in the input depth map which correspond to each of the selected pixels; means for determining weights for each pixel within the window region based at least on a difference between a depth value corresponding to each of the selected pixels and the determined reference depth value for the image pixel; means for determining a refined depth value for the image pixel from a weighted sum of the depth values corresponding to each of the selected pixels, said depth values being weighted in accordance with the determined weights; and means for determining a high resolution depth map corresponding to the image from the refined depth value.

13 . A non-transitory computer readable medium having a computer program stored thereon for determining a high resolution depth map corresponding to an image of a scene, said program comprising:
code for receiving the image and an input depth map corresponding to the image; code for selecting, from a window region about an image pixel of the image, a plurality of pixels having a substantially similar characteristic value as the image pixel; code for determining a reference depth value for the image pixel from depth values in the input depth map which correspond to each of the selected pixels; code for determining weights for each pixel within the window region based at least on a difference between a depth value corresponding to each of the selected pixels and the determined reference depth value for the image pixel; code for determining a refined depth value for the image pixel from a weighted sum of the depth values corresponding to each of the selected pixels, said depth values being weighted in accordance with the determined weights; and code for determining a high resolution depth map corresponding to the image from the refined depth value.

Assignees

• Canon Kk

Inventors

• Wan Ernest Yiu Cheong

Classifications

• G06T3/40 Primary

  Scaling of whole images or parts thereof, e.g. expanding or contracting · CPC title

• G06K9/6215

  Physics · mapped topic

• G06T7/0051 Primary

  Physics · mapped topic

• G06K2009/4666

  Physics · mapped topic