Method and apparatus for converting depth image in high resolution

Patent metadata

Field	Value
Publication number	US-9613293-B2
Application number	US-201214361467-A
Country	US
Kind code	B2
Filing date	Nov 29, 2012
Priority date	Nov 29, 2011
Publication date	Apr 4, 2017
Grant date	Apr 4, 2017

Abstract

Provided are a method and an apparatus for image processing. A scene includes a low resolution depth image and a high resolution color image. A depth image up-scaler up-scales the depth image using an interpolation so as to improve the resolution of the depth image. A color-depth similarity based filter removes noise from the up-scaled depth image using the color image and the up-scaled depth image. A boundary sharpening filter converts a boundary between objects in the up-scaled depth image into a clearer boundary. The resolution of the depth image may increase to the level of the resolution of the color image by means of the depth image up-scaler, the color-depth similarity based filter and the boundary sharpening filter, and the depth of each pixel in the depth image may be adjusted.

First claim

The invention claimed is:

1. A method, comprising:
increasing, using a processor, a depth image resolution of a depth image to match a color image resolution of a two dimensional color image of a same scene and to produce an up-scaled depth image;
removing, using the processor, noise in the up-scaled depth image using the color image and the up-scaled depth image, the removing including updating a depth of a target pixel in the up-scaled depth image based on similarities of neighboring pixels with respect to the target pixel and depths of the neighboring pixel, the similarities of the neighboring pixels being based on differences in one or more attribute values between the neighboring pixels and the target pixel;
sharpening, using the processor, a boundary of an object in the up-scaled depth image from which the noise is removed to produce sharpened depth image; and
generating, using the processor, a three dimensional color model of the scene using the sharpened depth image and the two dimensional color image.

2. The method of claim 1 , wherein the sharpening adjusts a pixel depth of a boundary pixel to a closest one of a maximum and a minimum pixel depth of neighboring pixels in the boundary.

3. The method of claim 1 , wherein
the neighboring pixels correspond to pixels, included in a window of which a center is the target pixel, from among pixels in the up-scaled depth image, and
the differences in the one or more attribute values including at least one of,
a first difference between a color of a pixel of the color image corresponding to the neighboring pixel and a color of a pixel of the color image corresponding to the target pixel,
a second difference between a depth of the neighboring pixel and a depth of the target pixel, and
a third difference between a location of the neighboring pixel and a location of the target pixel.

4. A method of processing a depth image and a color image associated with a scene, the method comprising:
increasing, using a processor, a depth image resolution of the depth image by up-scaling the depth image using interpolation to produce an up-scaled depth image;
removing, using the processor, noise in the up-scaled depth image using the color image and the up-scaled depth image, the removing including updating a depth of a target pixel in the up-scaled depth image based on similarities of neighboring pixels with respect to the target pixel and depths of the neighboring pixels, the similarities of the neighboring pixels being based on differences in one or more attribute values between the neighboring pixels and the target pixel; and
sharpening, using the processor, a boundary between objects in the upscaled depth image from which the noise is removed,
wherein the depth image resolution of the up-scaled depth image matches a color image resolution of the color image.

5. The method of claim 4 , wherein
the neighboring pixels correspond to pixels included in a window of which a center is the target pixel, from among pixels in the up-scaled depth image, and
the differences in the one or more attribute values including at least one of,
a first difference between a color of a pixel of the color image corresponding to the neighboring pixel and a color of a pixel of the color image corresponding to the target pixel,
a second difference between a depth of the neighboring pixel and a depth of the target pixel, and
a third difference between a location of the neighboring pixel and a location of the target pixel.

6. The method of claim 5 , wherein the updating updates the depth of the target pixel to a value obtained by dividing a first result value by a second result value, wherein
the first result value is a sum of products of the depths of the neighboring pixels and the similarities of the neighboring pixels with respect to the target pixel, and
the second result value is a sum of the similarities of the neighboring pixels with respect to the target pixel.

7. The method of claim 5 , wherein
the pixels in the up-scaled depth image are sequentially selected as the target pixel and updated, and
the target pixel is repeatedly updated, at least once, based on previously updated depths of neighboring pixels.

8. The method of claim 5 , wherein the updating updates the target pixel within a range between a minimum value and a maximum value, the minimum value being a minimum depth from among depths of the neighboring pixels before updating, and the maximum value being a maximum depth from among the depths of neighboring pixels before updating.

9. The method of claim 5 , further comprising:
generating, using the processor, the color image by down-sampling an original color image associated with the scene to match the depth image resolution of the depth image,
wherein the increasing of the depth image resolution, the generating of the color image, and the removing of the noise are repeatedly performed until the depth image resolution of the up-scaled depth image matches an original resolution of the original color image.

10. The method of claim 4 , wherein the sharpening further comprises:
identifying a maximum depth and a minimum depth from among depths of neighboring pixels of a target pixel in the up-scaled depth image;
determining whether the target pixel exists in a boundary area between objects in the up-scaled depth image; and
replacing a depth of the target pixel with a replacement depth that is closer to the depth of the target pixel between the maximum depth and the minimum depth, if the target pixel exists in the boundary area,
wherein the neighboring pixels correspond to pixels included in a window of which a center is the target pixel, from among pixels in the up-scaled depth image.

11. The method of claim 10 , wherein the determining includes determining if a difference between the maximum depth and the minimum depth is greater than a threshold value.

12. A non-transitory computer-readable medium comprising computer-readable instructions for instructing a computer to perform the method of claim 4 .

13. An apparatus for processing a depth image and a color image associated with a scene, the apparatus comprising:
a processor; and
a memory containing computer-readable code that, when executed by the processor, causes the processor to,
perform up-scaling of the depth image using interpolation to increase a depth image resolution of the depth image to produce an up-scaled depth image,
remove noise in the up-scaled depth image using the color image and the up-scaled depth image by updating a depth of a target pixel in the up-scaled depth image based on similarities of neighboring pixels with respect to the target pixel and depths of the neighboring pixels, the similarities of the neighboring pixels being based on differences in one or more attribute values between the neighboring pixels and the target pixel, and
sharpen a boundary between objects in the up-scaled depth image from which the noise is removed,

wherein the depth image resolution of the up-scaled depth image matches a color image resolution of the color image.

14. The apparatus of claim 13 , wherein
the neighboring pixels correspond to pixels, included in a window of which a center is the target pixel, from among pixels in the up-scaled depth image, and
the differences in the one or more attribute values including at least one of,
a first difference between a color of a pixel of the color image corresponding to the neighboring pixel and a color of a pixel of the color image corresponding to the target pixel,
a second difference between a depth of the neighboring pixel and a depth of the target pixel, and

Assignees

• Choi Ouk
• Kim Do Kyoon
• Lee Kee Chang
• Samsung Electronics Co Ltd

Inventors

• Choi Ouk
• Kim Do Kyoon
• Lee Kee Chang

Classifications

• G06F18/22

  Matching criteria, e.g. proximity measures · CPC title

• G06T5/00

  Image enhancement or restoration · CPC title

• G06T7/408

  Physics · mapped topic

• G06K2009/4666

  Physics · mapped topic

• G06T7/0051

  Physics · mapped topic