Method and apparatus for segmentation of 3D image data

Patent metadata

Field	Value
Publication number	US-10096116-B2
Application number	US-201214766687-A
Country	US
Kind code	B2
Filing date	Dec 12, 2012
Priority date	Dec 12, 2012
Publication date	Oct 9, 2018
Grant date	Oct 9, 2018

Abstract

The present invention provides a method and an apparatus for real time object segmentation of 3D image data based on local feature correspondences between a plurality of views. In order to reduce the computational effort of object segmentation of 3D image data, the segmentation process is performed based on correspondences relating to local features of the image data and a depth map. In this way, computational effort can be significantly reduced and the image segmentation can be carried out very fast.

First claim

What is claimed is:

1. A method for generating a depth-based segmentation of a three dimensional (3D) image data of a 3D image, the method comprising:
determining a first set of local features within a first image data for a first view of the 3D image;
determining a second set of local features within a second image data for a second view of the 3D image;
determining a set of corresponding local features between the first set of local features of the first image data and the second set of local features of the second image data of the 3D image data; and
segmenting each of the first image data and the second image data into a plurality of corresponding depth regions of the first image data and second image data for the 3D image based on:
the set of corresponding local features within corresponding depth regions of the first image data and the second image data, and
a depth map of the 3D image,

wherein the depth map is made up of depth map elements defined by an image position and a depth value,
wherein each local feature describes a patch of the 3D image surrounding a point within the 3D image, and
wherein the segmenting comprises:
identifying a set of initial depth regions, wherein each initial depth region is:
a group of contiguous depth map elements, and
formed from the depth map based upon the depth values assigned to individual ones of the depth map elements;

computing, for pairs of corresponding initial depth regions in the first image data and the second image data, an average spatial displacement of corresponding local features within a pair of corresponding initial depth regions; and
merging adjacent ones of the set of initial depth regions based on similarity between the adjacent ones of the initial depth regions as measured by a correlation value for the adjacent ones of the initial depth regions,
wherein the correlation value is based upon both:
respective depth values assigned to the adjacent ones of the initial depth regions, and
respective average spatial displacement of corresponding local features within the pair of corresponding initial depth regions, computed during the computing, for the adjacent ones of the initial depth regions.

2. The method according to claim 1 , wherein the segmenting comprises quantizing the depth value of the depth map elements of the depth map of the 3D image; and
wherein the identifying a set of initial depth regions comprises identifying the initial depth regions by determining contiguous depth map elements having a same quantized depth value.

3. The method according to claim 1 , wherein the segmenting further comprises:
eliminating erroneous local feature correspondences having a value of the spatial displacement larger than the predetermined displacement value.

4. The method according to claim 1 , wherein the segmenting further comprises:
generating a graph having vertexes corresponding to the determined local features; and
applying a graph cut process to the generated graph to obtain auxiliary image segmentation;
wherein the adjacent depth regions are merged using the obtained auxiliary image segmentation.

5. The method according to claim 1 , the segmenting further comprising:
identifying an edge extending through a depth region of the plurality of corresponding depth regions; and
segmenting the depth region into at least two depth regions according to the identified edge.

6. The method according to claim 1 , the segmenting further comprising segmenting the 3D image data into foreground image data and background image data, wherein the segmentation is performed only for the foreground image data.

7. The method according to claim 1 , wherein the 3D image data are obtained from a video sequence.

8. The method according to claim 1 , wherein the local feature correspondences are determined by a matching process comparing the determined local features of the first view and the second view of the 3D image.

9. The method according claim 1 , wherein the value of the spatial displacement of corresponding local features is determined with respect to a single spatial direction.

10. An apparatus for generating a depth-based segmentation of a three dimensional (3D) image data of a 3D image, the apparatus comprising:
a receiver;
a processor; and
a non-transitory computer-readable medium including computer-executable instructions,
wherein the receiver is configured to receive 3D image data from a 3D image data source,
wherein the 3D image data comprises a first image data for a first view and a second image data for a second view, and
wherein the processor is configured to execute the computer-executable instructions on the non-transitory computer-readable medium to carry out generating the depth-based segmentation of the 3D image data according to a method comprising:
determining a first set of local features within a first image data for a first view of the 3D image;
determining a second set of local features within a second image data for a second view of the 3D image;
determining a set of corresponding local features between the first set of local features of the first image data and the second set of local features of the second image data of the 3D image data; and
segmenting each of the first image data and the second image data into a plurality of corresponding depth regions of the first image data and second image data for the 3D image based on:
the set of corresponding local features within corresponding depth regions of the first image data and the second image data, and
a depth map of the 3D image,

wherein the depth map is made up of depth map elements defined by an image position and a depth value,
wherein each local feature describes a patch of the 3D image surrounding a point within the 3D image, and
wherein the segmenting comprises:
identifying a set of initial depth regions, wherein each initial depth region is:
a group of contiguous depth map elements, and
formed from the depth map based upon the depth values assigned to individual ones of the depth map elements;

computing, for pairs of corresponding initial depth regions in the first image data and the second image data, an average spatial displacement of corresponding local features within a pair of corresponding initial depth regions; and
merging adjacent ones of the set of initial depth regions based on similarity between the adjacent ones of the initial depth regions as measured by a correlation value for the adjacent ones of the initial depth regions,
wherein the correlation value is based upon both:
respective depth values assigned to the adjacent ones of the initial depth regions, and
respective average spatial displacement of corresponding local features within the pair of corresponding initial depth regions, computed during the computing, for the adjacent ones of the initial depth regions.

11. The apparatus according to claim 10 , wherein the segmenting comprises quantizing the depth value of the depth map elements of the depth map of the 3D image; and
wherein the identifying a set of initial depth regions comprises identifying the initial depth regions by determining contiguous depth map elements having a same quantized depth value.

12. The apparatus according to claim 11 , wherein the segmenting further comprises:
eliminating erroneous local feature correspondences having a value of the spatial displacement larger than the predetermined displacement value.

13. The apparatus according to claim 10 , wherein the segmenting further comprises:
generating a graph having vertexes corresponding to the determined local features; and
applying a graph cut process to the

Assignees

• Huawei Tech Co Ltd

Inventors

• Cordara Giovanni
• Bouazizi Imed
• Kondrad Lukasz

Classifications

• G06T2207/10028

  Range image; Depth image; 3D point clouds · CPC title

• G06T7/11 Primary

  Region-based segmentation · CPC title

• H04N2013/0081

  Depth or disparity estimation from stereoscopic image signals · CPC title

• G06T2207/10012

  Stereo images · CPC title

• H04N13/128 Primary

  Adjusting depth or disparity · CPC title