Method and apparatus for refining depth image

Patent metadata

Field	Value
Publication number	US-10991111-B2
Application number	US-201815913055-A
Country	US
Kind code	B2
Filing date	Mar 6, 2018
Priority date	Apr 5, 2017
Publication date	Apr 27, 2021
Grant date	Apr 27, 2021

Abstract

A method of refining a depth image includes extracting shading information of color pixels from a color image, and refining a depth image corresponding to the color image based on surface normal information of an object included in the shading information.

First claim

What is claimed is:

1. A method of refining a depth image, the method comprising:
extracting shading information of color pixels from a color image; and
refining a depth image corresponding to the color image based on surface normal information of an object included in the shading information, wherein the refining of the depth image includes distinguishing between a noise region and an edge region in the depth image based on a characteristic of a surface normal distribution represented in the color image and applying filters comprising different filter characteristics to the noise region and the edge region respectively to reduce a noise component while maintaining an edge characteristic substantially unchanged.

2. The method of claim 1 , further comprising extracting shading information of the color pixels based on depth information of the refined depth image.

3. The method of claim 1 , wherein the shading information corresponds to a vector dot product between the direction of the light source and the surface normal of the object surface.

4. A non-transitory computer-readable medium storing instructions that, when executed by a processor, cause the processor to perform the method of claim 1 .

5. The method of claim 1 , wherein the refining of the depth image comprises:
determining the filter characteristic of the filter to be applied to a current depth pixel, of the depth pixels, included in the depth image based on a surface normal distribution of the surface normal information for each of regions in the depth image; and
adjusting a depth value of the current depth pixel by applying the filter having the determined filter characteristic to the current depth pixel.

6. The method of claim 5 , wherein the determining of the filter characteristic comprises determining either one or both of a filter coefficient of the filter to be applied to the current depth pixel and a filter size of the filter to be applied to the current depth pixel based on the surface normal information.

7. The method of claim 1 , further comprising determining whether to refine a depth image of a current time based on depth information of a depth image of a previous time.

8. The method of claim 7 , further comprising refining the depth image of the current time based on the depth information of the depth image of the previous time in response to a difference between a color image corresponding to the depth image of the current time and a color image corresponding to the depth image of the previous time satisfying a preset condition.

9. A method of refining a depth image, the method comprising:
extracting shading information of color pixels from a color image; and
refining a depth image corresponding to the color image based on surface normal information of an object included in the shading information, wherein the refining of the depth image comprises:
determining a type of a region to which a current depth pixel included in the depth image belongs based on a surface normal distribution of the surface normal information in the region; and
adjusting a depth value of the current depth pixel by applying a filter corresponding to the determined type of the region to the current depth pixel.

10. The method of claim 9 , wherein the filter is configured to adjust the depth value of the current depth pixel based on a depth value of a neighboring depth pixel of the current depth pixel.

11. The method of claim 9 , wherein the determining of the type of the region comprises determining a region to which the current depth pixel belongs based on a change of surface normal values of neighboring pixels of the current depth pixel.

12. The method of claim 9 , wherein the determining of the type of the region comprises determining whether the current depth pixel belongs to a noise region, a surface region of the object, or an edge region of the object.

13. The method of claim 12 , wherein a filter size of a filter corresponding to the noise region or the surface region of the object is greater than a filter size of a filter corresponding to the edge region of the object.

14. The method of claim 12 , wherein a filter coefficient to be applied to a neighboring depth pixel of the current depth pixel varies depending on whether the filter to be applied to the current depth pixel is a filter corresponding to the noise region, a filter corresponding to the surface region of the object, or a filter corresponding to the edge region of the object.

15. A method of refining a depth image, the method comprising:
extracting shading information of color pixels from a color image; and
refining a depth image corresponding to the color image based on surface normal information of an object included in the shading information, further comprising extracting albedo information of the color pixels from the color image;
wherein the refining of the depth image comprises refining the depth image based on a first weight based on the surface normal information, a second weight based on the albedo information, and a third weight based on a difference between a color image of a current time and a color image of a previous time.

16. A depth image refining apparatus comprising:
a processor configured to:
extract shading information of color pixels from a color image, and
refine a depth image corresponding to the color image based on surface normal information of an object included in the shading information,
wherein the refining of the depth image includes distinguishing between a noise region and an edge region in the depth image based on a characteristic of a surface normal distribution represented in the color image and applying filters comprising different filter characteristics to the noise region and the edge region respectively to reduce a noise component while maintaining an edge characteristic substantially unchanged.

17. The depth image refining apparatus of claim 16 , wherein the processor is further configured to:
determine a filter characteristic of the filter to be applied to a current depth pixel, of the depth pixels, included in the depth image based on a surface normal distribution of the surface normal information for each of regions in the depth image, and
adjust a depth value of the current depth pixel by applying the filter having the determined filter characteristic to the current depth pixel.

18. The depth image refining apparatus of claim 16 , wherein the processor is further configured to:
determine a type of a region to which a current depth pixel, of the depth pixels, included in the depth image belongs based on a surface normal distribution of the surface normal information of the region, and
adjust a depth value of the current depth pixel by applying a filter corresponding to the determined type of the region to the current depth pixel.

19. The depth image refining apparatus of claim 16 , wherein the processor is further configured to:
extract albedo information of the color pixels from the color image, and
refine the depth image based on a first weight based on the surface normal information, a second weight based on the albedo information, and a third weight based on a difference between a color image of a current time and a color image of a previous time.

20. A method of refining a depth image, the method comprising:
determining a noise reducing method to be applied to a depth image based on surface normal information of an object in a color image corresponding to the depth image; and
refining the depth image by applying the determined no

Assignees

• Samsung Electronics Co Ltd

Inventors

• Ha Inwoo
• Lee Hyong Euk
• Ahn Minsu
• Sung Young Hun

Classifications

• G06T7/507 Primary

  from shading (G06T7/586 takes precedence) · CPC title

• G06V20/653

  by matching three-dimensional models, e.g. conformal mapping of Riemann surfaces · CPC title

• G06T2207/20024

  Filtering details · CPC title

• G06T2207/10028

  Range image; Depth image; 3D point clouds · CPC title

• G06T7/90

  Determination of colour characteristics · CPC title