Method and apparatus for generating depth information

Patent metadata

Field	Value
Publication number	US-9406140-B2
Application number	US-201414307506-A
Country	US
Kind code	B2
Filing date	Jun 18, 2014
Priority date	Apr 17, 2014
Publication date	Aug 2, 2016
Grant date	Aug 2, 2016

Abstract

A method and an apparatus for generating depth information are provided. A main depth map associated with one of a left image and a right image and corresponding to multiple first pixels is obtained. The left image or the right image associated with the main depth map is divided into multiple segments according to pixel information, so as to obtain a segment distribution map including the segments. Multiple invalid depth values which do not match to a reliable condition are removed from the main depth map according to the segment distribution map, so as to generate a necessary repair depth map including multiple holes. Multiple optimized depth values are respectively generated for the holes in the necessary repair depth map according to the segment distribution map, and the optimized depth values are filled into the necessary repair depth map to generate an optimized depth map.

First claim

What is claimed is:

1. A method for generating depth information, for an electronic apparatus, the method comprising:
obtaining a left image and a right image captured by a stereoscopic imaging system and performing three-dimensional (3D) depth estimation on the left image and the right image so as to obtain a main depth map associated with one of the left image and the right image and corresponding to a plurality of first pixels, wherein the main depth map records a plurality of main depth values respectively corresponding to the first pixels;
dividing the left image or the right image associated with the main depth map into a plurality of segments according to pixel information of each of the first pixels, so as to obtain a segment distribution map including the segments;
determining whether the main depth values corresponding to the first pixels match a reliable condition according to the segment distribution map and removing a plurality of invalid depth values among the main depth values which do not match the reliable condition from the main depth map, so as to generate a necessary repair depth map including a plurality of holes, comprising:
dividing the main depth map into a plurality of depth segments according to the segment distribution map and performing a statistical calculation on the main depth values in each of the depth segments to obtain a statistical result; and
determining whether the main depth values are a plurality of first invalid depth values among the invalid depth values according to the statistical result and removing the first invalid depth values from the main depth map, so as to generate the necessary repair depth map including a plurality of first holes among the holes; and

generating a plurality of optimized depth values respectively for the holes in the necessary repair depth map according to the segment distribution map and filling the optimized depth values into the necessary repair depth map to generate an optimized depth map.

2. The method according to claim 1 , wherein the step of dividing the left image or the right image associated with the main depth map into the segments according to the pixel information of each of the first pixels so as to obtain the segment distribution map including the segments comprises:
comparing a first adjacent pixel with a second adjacent pixel that are adjacent to each other among the first pixels to determine whether a pixel value difference between the first adjacent pixel and the second adjacent pixel is less than a difference threshold; and
if the pixel value difference is less than the difference threshold, connecting the first adjacent pixel with the second adjacent pixel to form a first segment among the segments, wherein the first segment comprises the first adjacent pixel and the second adjacent pixel.

3. The method according to claim 2 , wherein the step of dividing the left image or the right image associated with the main depth map into the segments according to the pixel information of each of the first pixels so as to obtain the segment distribution map including the segments comprises:
limiting a segment size of each of the segments according to a size threshold to obtain the segment distribution map, wherein the segment size of each of the segments is not greater than the size threshold.

4. The method according to claim 1 , wherein after the step of removing the first invalid depth values from the main depth map so as to generate the necessary repair depth map including the first holes among the holes, the method further comprises:
calculating a plurality of invalid density values of the first invalid depth values within a limited range respectively by using a plurality of first valid depth values which are not the first invalid depth values among the main depth values as centers;
determining whether the first valid depth values are a plurality of second invalid depth values among the invalid depth values respectively according to whether the invalid density values are greater than a density threshold; and
removing the second invalid depth values from the main depth map, so as to generate the necessary repair depth map including a plurality of second holes among the holes.

5. The method according to claim 1 , further comprising:
performing the 3D depth estimation on the left image and the right image so as to obtain an auxiliary depth map associated with the other of the left image and the right image and corresponding to a plurality of second pixels, wherein the auxiliary depth map records an auxiliary depth value of each of the second pixels; and
comparing the main depth map with the auxiliary depth map to remove the main depth values which is not consistent with the corresponding auxiliary depth values thereof from the main depth map, so as to generate the necessary repair depth map including a plurality of third holes among the holes.

6. The method according to claim 3 , wherein the segment distribution map has a segmentation fineness according to the difference threshold and the size threshold, the segment distribution map comprises a first segment distribution map and a second segment distribution map, and the segmentation fineness of the first segment distribution map is different from the segmentation fineness of the second segment distribution map.

7. The method according to claim 6 , wherein the step of generating the optimized depth values respectively for the holes in the necessary repair depth map according to the segment distribution map and filling the optimized depth values into the necessary repair depth map to generate the optimized depth map comprises:
dividing the necessary repair depth map into a plurality of first depth segments according to the first segment distribution map and obtaining a first valid density value according to the number of the holes in each of the first depth segments;
determining whether to calculate a first average depth value of each of the first depth segments according to the first valid density value and filling part of the holes by using the first average depth value as one of the optimized depth values; and
repeating the preceding two steps until the times of repeating the preceding two steps reach a predetermined number.

8. The method according to claim 7 , wherein the step of determining whether to calculate the first average depth value of each of the first depth segments according to the first valid density value comprises:
determining whether to calculate the first average depth value of each of the first depth segments according to whether the first valid density value is greater than a valid threshold.

9. The method according to claim 7 , wherein the step of determining whether to calculate the first average depth value of each of the first depth segments according to the first valid density value further comprises:
obtaining a processing sequence of each of the first depth segments according to the first valid density value of each of the first depth segments and determining whether to fill the holes in each of the first depth segments according to whether the processing sequence has a high priority.

10. The method according to claim 7 , wherein after the step of determining whether to calculate the first average depth value of each of the first depth segments according to the first valid density value and filling part of the holes by using the first average depth value as one of the optimized depth values, the method further comprises:
dividing the necessary repair depth map into a plurality of second depth segments according to the second segment distribution map and obtaining a second valid density value according to the number of the holes in each of the se

Assignees

• Altek Semiconductor Corp

Inventors

• Lee Yun-Chin
• Tseng Chia-Chun
• Chung Yu-Chia

Classifications

• G06T2207/10012

  Stereo images · CPC title

• G06T7/0075 Primary

  Physics · mapped topic

• H04N13/0018

  Electricity · mapped topic

• G06T7/0081

  Physics · mapped topic

• G06T5/005

  Physics · mapped topic