Method and apparatus for scene segmentation from focal stack images

Patent metadata

Field	Value
Publication number	US-9401027-B2
Application number	US-201314058819-A
Country	US
Kind code	B2
Filing date	Oct 21, 2013
Priority date	Oct 21, 2013
Publication date	Jul 26, 2016
Grant date	Jul 26, 2016

Abstract

A method, apparatus and computer program product are provided to facilitate scene segmentation from focal stack images. The method may include receiving a set of focal stack images, calculating a focal measure for each of a plurality of pixels of the set of focal stack images, and grouping each of a plurality of pixels for which the focal measure was calculated into a plurality of super pixels. The method may also include calculating a focal measure for each of the plurality of super pixels, segmenting a respective focal stack image based on the focal measure of each of the plurality of super pixels, calculating a color probability for each of the plurality of super pixels, and segmenting each focal stack image based on color probability of each of the plurality of super pixels.

First claim

That which is claimed:

1. A method to facilitate scene segmentation of focal stack images, the method comprising:
receiving a set of focal stack images, wherein a focal stack image comprises a plurality of pixels;
grouping each of the plurality of pixels to which an over segmentation algorithm or a decimation algorithm is applied based on a shape of the plurality of pixels to create a plurality of super pixels;
calculating a focal measure for each of the plurality of the super pixels;
segmenting a respective focal stack image based on the focal measures of the plurality of super pixels;
calculating a color probability for respective super pixels; and
segmenting the respective focal stack image based on the color probability of respective super pixels.

2. A method according to claim 1 further comprising:
selecting a representative plurality of focal images from the set of focal stack images.

3. A method according to claim 1 further comprising:
aligning the set of focal stack images.

4. A method according to claim 1 further comprising:
receiving a selection of a focus frame;
wherein the segmenting the respective focal stack image based on the focal measure is further based on the selected focus frame; and
wherein the segmenting the respective focal stack image based on color probability is further based on the selected focus frame.

5. A method according to claim 1 further comprising:
classifying the plurality of super pixels as foreground or background.

6. A method according to claim 5 further comprising:
shrinking segmentation mask consisting of multiple super pixels based on classification as background or foreground.

7. A method according to claim 5 further comprising:
removing island pixels based on classification as background or foreground; and
setting matting layers based on pixel classification as background or foreground.

8. An apparatus to facilitate scene segmentation of focal stack images, the apparatus comprising a processor and a memory including computer program code, the memory and computer program code configured to, with the processor, cause the apparatus to:
receive a set of focal stack images, wherein a focal stack image comprises a plurality of pixels;
group each of the plurality of pixels to which an over segmentation algorithm or a decimation algorithm is applied based on a shape of the plurality of pixels to create a plurality of super pixels;
calculate a focal measure for each of the plurality of the super pixels;
segment a respective focal stack image based on the focal measures of the plurality of super pixels;
calculate a color probability for respective super pixels; and
segment the respective focal stack image based on the color probability of respective super pixels.

9. An apparatus according to claim 8 wherein the memory and computer program code are further configured to, with the processor, cause the apparatus to:
select a representative plurality of focal images from the set of focal stack images.

10. An apparatus according to claim 8 wherein the memory and computer program code are further configured to, with the processor, cause the apparatus to:
align the set of focal stack images.

11. An apparatus according to claim 8 wherein the memory and computer program code are further configured to, with the processor, cause the apparatus to:
receive a selection of a focus frame;
wherein the segmenting the respective focal stack image based on the focal measure is further based on the selected focus frame; and
wherein the segmenting the respective focal stack image based on color probability is further based on the selected focus frame.

12. An apparatus according to claim 8 wherein the memory and computer program code are further configured to, with the processor, cause the apparatus to:
classify the plurality of super pixels as foreground or background.

13. An apparatus according to claim 12 wherein the memory and computer program code are further configured to, with the processor, cause the apparatus to:
shrink segmentation mask consisting of multiple super pixels based on classification as background or foreground.

14. An apparatus according to claim 12 wherein the memory and computer program code are further configured to, with the processor, cause the apparatus to:
remove island pixels based on classification as background or foreground; and
set matting layers based on pixel classification as background or foreground.

15. A computer program product to facilitate scene segmentation of focal stack images, the computer program product comprising a non-transitory computer readable medium having program code portions stored thereon, the program code portions configured, upon execution to:
receive a set of focal stack images;
select a representative plurality of focal images from the set of focal stack images, wherein a focal stack image comprises a plurality of pixels;
group each of the plurality of pixels to which an over segmentation algorithm or a decimation algorithm is applied based on a shape of the plurality of pixels to create a plurality of super pixels;
calculate a focal measure for each of the plurality of the super pixels;
segment a respective focal stack image based on the focal measures of the plurality of super pixels;
calculate a color probability for respective super pixels; and
segment the respective focal stack image based on the color probability of respective super pixels.

16. A computer program product according to claim 15 wherein the program code portions are further configured, upon execution to
align the set of focal stack images.

17. A computer program product according to claim 15 wherein the program code portions are further configured, upon execution, to
receive a selection of a focus frame;
wherein the segmenting the respective focal stack image based on the focal measure is further based on the selected focus frame; and
wherein the segmenting the respective focal stack image based on color probability is further based on the selected focus frame.

18. A computer program product according to claim 15 wherein the program code portions are further configured, upon execution, to
classify the plurality of super pixels as foreground or background.

19. A computer program product according to claim 18 wherein the program code portions are further configured, upon execution, to
shrink segmentation mask consisting of multiple super pixels based on classification as background or foreground.

20. A computer program product according to claim 18 wherein the program code portions are further configured, upon execution, to
remove island pixels based on classification as background or foreground; and
set matting layers based on pixel classification as background or foreground.

21. The method of claim 1 further comprising:
calculating the focal measure for each of a plurality of pixels of the set of focal stack images.

22. An apparatus according to claim 8 wherein the memory and computer program code are further configured to, with the processor, cause the apparatus to:
calculate the focal measure for each of a plurality of pixels of the set of focal stack images.

23. A computer program product according to claim 15 wherein the program code portions are further configured, upon execution, to
calculate the focal measure for each of a plurality of pixels of the set of focal stack images.

Assignees

• Nokia Technologies Oy

Inventors

• Zhou Hui
• Ahonen Timo
• Benchemsi Karim
• Karlsson Tobias
• Mustajbasic Amer
• Windmark Johan

Classifications

• G06T2207/10024

  Color image · CPC title

• G06T7/0097

  Physics · mapped topic

• G06T2207/20144

  Physics · mapped topic

• G06T2207/20104

  Interactive definition of region of interest [ROI] · CPC title

• G06T7/0093

  Physics · mapped topic