Object segmentation from light field data

Patent metadata

Field	Value
Publication number	US-10136116-B2
Application number	US-201615253037-A
Country	US
Kind code	B2
Filing date	Aug 31, 2016
Priority date	Mar 7, 2016
Publication date	Nov 20, 2018
Grant date	Nov 20, 2018

Abstract

A scene is segmented into objects based on light field data for the scene, including based on image pixel values (e.g., intensity) and disparity map(s). In one aspect, the light field data is used to estimate one or more disparity maps for the scene taken from different viewpoints. The scene is then segmented into a plurality of regions that correspond to objects in the scene. Unlike other approaches, the regions can be variable-depth. In one approach, the regions are defined by boundaries. The boundaries are determined by varying the boundary to optimize an objective function for the region defined by the boundary. The objective function is based in part on a similarity function that measures a similarity of image pixel values for pixels within the boundary and also measures a similarity of disparities for pixels within the boundary.

First claim

What is claimed is:

1. A computer-implemented method for segmenting a scene into objects based on light field data for the scene, the method implemented on a processor executing instructions and the method comprising:
accessing light field data for the scene, the light field data including a plurality of images of the scene taken from different viewpoints, the images comprising pixels having image pixel values;
from the light field data, estimating one or more disparity maps for the scene taken from different viewpoints; and
segmenting the scene into variable-depth regions that correspond to objects in the scene, the variable-depth regions defined by boundaries, at least one boundary of the boundaries defining at least one variable-depth region of the variable-depth regions determined by varying the at least one boundary to optimize an objective function for the at least one-variable-depth region; wherein:
the objective function for the at least one variable-depth region is based in part on a similarity function that measures a similarity of image pixel values for pixels within the at least one boundary defining the at least one variable-depth region;
the similarity function takes a form of

S
⁡

(

x
,
L

)

=

[

D
⁡

(

x
,

p
x

)

-

1


L


⁢

∑

x
∈
L

⁢

D
⁡

(

x
,

p
x

)

]

2

 where x denotes a pixel in the image, x is a disparity of that pixel x, D(x, x ) is a descriptor function, L is the at least one variable-depth region within the at least one boundary, and |L| denotes a cardinality of region L; and
the descriptor function D(x, x ) is based on:
[μ( x, x )] 2

 where μ(x, x ) is a mean image pixel value for all pixels along an epipolar line that passes through the pixel x with the associated disparity x .

2. The method of claim 1 wherein the similarity function also measures a similarity of disparities for pixels within the at least one variable-depth region defined by the at least one boundary.

3. The method of claim 1 wherein the objective function is further based in part on a second similarity function that measures a similarity of pixels outside the at least one variable-depth region defined by the at least one boundary.

4. The method of claim 3 wherein the similarity function for pixels within the at least one variable-depth region has a same functional form as the similarity function for pixels outside the at least one variable-depth region.

5. The method of claim 1 wherein the objective function is further based in part on a factor that measures a length of the at least one boundary defining the at least one variable-depth region.

6. The method of claim 1 wherein the descriptor function D(x, x ) is based on:
[ I ( x )−μ( x, x )] 2

where I(x) is the image pixel value for pixel x.

7. The method of claim 1 wherein the similarity function is based on a descriptor function that is calculated for a plurality of different viewpoints.

8. The method of claim 1 wherein the similarity function is based on a descriptor function that is calculated for a plurality of different color channels.

9. The method of claim 1 wherein varying the at least one boundary to optimize an objective function for the at least one variable-depth region comprises:
evaluating the similarity function for an initial boundary for the at least one boundary and evolving the at least one boundary based on an active contour framework.

10. The method of claim 1 wherein estimating one or more disparity maps comprises estimating the disparity maps based on epipolar structure in the light field data.

11. The method of claim 1 wherein estimating one or more disparity maps accounts for occlusions in the scene.

12. The method of claim 1 wherein estimating one or more disparity maps is based on a Ray-Gaussian transform.

13. The method of claim 1 further comprising:
capturing the light field data for the scene before accessing the light field data.

14. The method of claim 1 wherein the light field data is light field data for an interior of an ear canal, and segmenting the scene into a plurality of regions comprises segmenting the scene into a tympanic membrane region and a background region.

15. A computer-implemented method for segmenting a scene into objects based on light field data for the scene, the method implemented on a processor executing instructions and the method comprising:
accessing light field data for the scene, the light field data including a plurality of images of the scene taken from different viewpoints, the images comprising pixels having image pixel values;
from the light field data, estimating one or more disparity maps for the scene taken from different viewpoints; and
segmenting the scene into variable-depth regions that correspond to objects in the scene, the variable-depth regions defined by boundaries, at least one boundary of the boundaries defining at least one variable-depth region of the variable-depth regions determined by varying the at least one boundary to optimize an objective function for the at least one variable-depth region; wherein:
the objective function for the at least one variable-depth region is based in part on a similarity function that measures a similarity of image pixel values for pixels within the at least one boundary defining the at least one variable-depth region;
the similarity function takes a form of

S
⁡

(

x
,

Assignees

• Tosic Ivana
• Kuruvilla Anupama
• Berkner Kathrin
• Kovacevic Jelena
• Ricoh Co Ltd
• Univ Carnegie Mellon

Inventors

• Tosic Ivana
• Kuruvilla Anupama
• Berkner Kathrin
• Kovacevic Jelena

Classifications

• G06T7/11

  Region-based segmentation · CPC title

• G06T2207/20016

  Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform · CPC title

• G06T2200/21

  involving computational photography · CPC title

• G06T7/557 Primary

  from light fields, e.g. from plenoptic cameras · CPC title

• G06T7/162

  involving graph-based methods · CPC title