Depth/disparity map post-processing method and device

Patent metadata

Field	Value
Publication number	US-10424075-B2
Application number	US-201515565877-A
Country	US
Kind code	B2
Filing date	May 6, 2015
Priority date	May 6, 2015
Publication date	Sep 24, 2019
Grant date	Sep 24, 2019

Abstract

A method and a device for post-processing depth/disparity maps adopt a strategy of combining edge information and segmentation information when detecting irregular edge regions. The method includes dividing a color image into super pixels when performing image segmentation on a color image; partitioning a grayscale range into a preset number of intervals, and for each super pixel, statistically obtaining a histogram of all the pixel points that fall within the intervals; determining, in a current super pixel, whether a ratio of the number of pixels contained in the interval having a maximum interval distribution value, to the total number of pixels in the current super pixel is less than the first threshold; and if so, further dividing the current super pixel using a color-based segmentation method. The disclosed method and device improve accuracy of color image division while ensuring image processing speed, thus improving detection accuracy of the irregular edge regions.

First claim

What is claimed is:

1. A method for post-processing depth/disparity maps, comprising:
inputting an image to be processed, wherein the image to be processed is a depth map or a disparity map;
extracting first edges from the depth map or the disparity map to be processed by an irregular edge detection module in a computer device;
segmenting a color image corresponding to the image to be processed to obtain segmentation information, wherein the step of segmenting a color image includes:
dividing the color image into super pixels;
partitioning a grayscale range into a preset number of intervals; and
for each super pixel, statistically obtaining a histogram of all the pixel points falling within the intervals,
wherein the segmentation information includes second edges, wherein a portion of the super pixels are in the second edges;

determining, in a current super pixel, whether a ratio of the number of pixels contained in the interval having a maximum interval distribution value, to a total number of pixels in the current super pixel is less than a first threshold; and if so, further dividing the current super pixel using color-based segmentation, wherein at least two of the subdivided super pixels have unequal sizes;
obtaining an irregular edge region in the image to be processed by comparing the first edges with the second edges, wherein the irregular edge region is identified when the first edges differ from the second edges; and
repairing the irregular edge region by an irregular edge repair module in the computer device.

2. The method of claim 1 , wherein a plurality of irregular edge regions are obtained, wherein the irregular edge regions include a protruding region and a concave region, wherein both the protruding region and the concave region are repaired by the irregular edge repair module.

3. The method of claim 1 , wherein the step of obtaining an irregular edge region in the image to be processed comprises:
determining the irregular edge region based on the edge information and segmentation information;
constructing a square window with its center at a point on an irregular edge, wherein the irregular edge divides the square window into several portions; and
assigning a portion with smallest area as the irregular edge region.

4. The method of claim 3 , wherein the step of repairing the irregular edge region comprises:
repairing the irregular edge region using a weighted median filter method using a guided filter coefficient as a filter kernel coefficient.

5. The method of claim 4 , wherein the guided filter coefficient is defined by:

W

p
,
q

⁡

(
I
)

=

1


w


2

⁢

∑

p
,

q
∈
w

⁢

(

1
+

(

I
p

-
u

)

⁢

(

I
q

-
u

)

∑

+
ɛ

⁢

⁢
U

)

where p and q are pixel points in the square window, |w| is the total number of pixels in the square window, I is the guided image, c is the smoothing coefficient, and U is the corresponding identity matrix,
wherein when the image to be processed is a disparity map, a binocular image pair is used as the guided map, I p and I q are 6D vectors, u is a 6D mean vector, and Σ is the 6*6 cross-correlation matrix,
wherein when the image to be processed is a depth map, a monocular image is used as the guided map, I p and I q are 3D vectors, u is a 3D mean vector, and Σ is 3*3 cross-correlation matrix.

6. The method of claim 1 , further comprising:
prior to the step of extracting edges from the image to be processed, detecting holes and filling the holes in the image to be processed, which comprises the steps of:
defining a pixel point having a disparity value less than a second threshold as a low confidence point;
for the low confidence point, when a disparity value of the low confidence point is less than any neighboring point by the second threshold, determining the low confidence point as a hole; repeating the steps of defining and determining to detect all holes in the image to be processed; and
when all the neighboring points are detected as non-holes, filling the current hole with a neighboring point having minimum disparity value; when any neighboring point is detected as a hole, filling the current hole with a neighboring point with the maximum disparity value; and repeating the step of filling the current hole to fill all holes in the image to be processed.

7. A depth/disparity map post-processing device, comprising:
an input module configured to input an image to be processed, wherein the image to be processed is a depth map or disparity map;
an irregular edge detection module comprising an edge extraction unit, an image segmentation unit, and an irregular edge detection unit,
wherein the edge extraction unit is configured to extract first edges from the depth map or the disparity map, wherein the image segmentation unit is configured conduct image segmentation of a color image corresponding to the image to be processed to obtain segmentation information, wherein the irregular edge detection unit is configured to obtain an irregular edge region in the image to be processed based on the edge information and the segmentation information,
wherein the image segmentation unit is configured to divide the color image into super pixels, to partition a grayscale range into a preset number of intervals, and to statistically obtain a histog

Assignees

• Univ Peking Shenzhen Graduate School

Inventors

• Jiao Jianbo
• Wang Ronggang
• Wang Zhenyu
• Wang Wenmin
• Gao Wen

Classifications

• G06T5/50 Primary

  using two or more images, e.g. averaging or subtraction · CPC title

• G06T7/50 Primary

  Depth or shape recovery · CPC title

• G06T7/136

  involving thresholding · CPC title

• G06T2200/04

  involving 3D image data · CPC title

• G06T2207/10028

  Range image; Depth image; 3D point clouds · CPC title