Method and system of background-foreground segmentation for image processing

Patent metadata

Field	Value
Publication number	US-9741125-B2
Application number	US-201514925057-A
Country	US
Kind code	B2
Filing date	Oct 28, 2015
Priority date	Oct 28, 2015
Publication date	Aug 22, 2017
Grant date	Aug 22, 2017

Abstract

Techniques for a system, article, and method of background-foreground segmentation for image processing may include obtaining pixel data including both non-depth data and depth data for at least one image, where the non-depth data includes color data or luminance data or both and associated with the pixels; determining whether a portion of the image is part of a background or foreground of the image based on the depth data and without using the non-depth data; and determining whether a border area between the background and foreground formed by using the depth data are part of the background or foreground depending on the non-depth data without using the depth data.

First claim

What is claimed is:

1. A computer-implemented method of background-foreground segmentation for image processing, comprising:
obtaining pixel data comprising both non-depth data and depth data for at least one image, wherein the non-depth data comprises color data or luminance data or both and associated with the pixels;
determining whether a portion of the image is part of a background or foreground of the image based on the depth data and without using the non-depth data; and
determining whether a border area between the background and foreground formed by using the depth data are part of the background or foreground depending on the non-depth data without using the depth data.

2. The method of claim 1 comprising determining whether at least one component established by detecting a plane in the image is in the background or foreground.

3. The method of claim 1 comprising performing over-segmentation on the image without using non-depth pixel data.

4. The method of claim 3 wherein the over-segmentation is performed on areas of the image other than those with components established by detecting planes.

5. The method of claim 3 comprising performing the over-segmentation by using a mean-shift algorithm.

6. The method of claim 1 comprising determining whether at least one component of the image is in the background or foreground depending on whether the component crosses a segmentation boundary established to indicate that segmentation of the image within the boundary is desired.

7. The method of claim 1 comprising merging components established by over-segmentation based on single depth values established for individual components.

8. The method of claim 1 comprising designating a first component as part of the foreground or background depending on (1) whether the component is the largest component on the image, or (2) whether the component has a largest or smallest depth relative to other individual components, or (3) both.

9. The method of claim 8 wherein the first component is a first foreground component, and when a group of large components of similar size exists on the image, the large component with the smallest depth relative to the depth of the other large components is the first component.

10. The method of claim 8 comprising determining whether other components are classified as background or foreground based, as least in part, on individual distances from the other components to the first component.

11. The method of claim 10 wherein the area of the closest other component of the remaining other components is added to the first component and classified as part of the background or foreground when the distance is below a maximum threshold distance.

12. The method of claim 10 wherein the distance is a 3D Euclidean distance.

13. The method of claim 10 wherein the distance is measured from closest edge of the other component to closest edge of the first component.

14. The method of claim 10 wherein the area of the first component is re-established after the area of each component is added to the first component before determining the distances to the remaining other separate components.

15. The method of claim 14 wherein the remaining other components are classified in the opposite segment of the first component when no remaining other component has a distance to the enlarged first component that is less than a maximum threshold distance.

16. The method of claim 1 comprising:
determining whether at least one component established by detecting a plane in the image is in the background or foreground;
performing over-segmentation on the image without using non-depth pixel data, wherein the over-segmentation is performed on areas of the image other than those with components established by detecting planes, and by using a mean-shift algorithm;
determining whether at least one component of the image is in the background or foreground depending on whether the component crosses a segmentation boundary established to indicate that segmentation of the image within the boundary is desired;
merging components established by over-segmentation based on single depth values established for individual components;
designating a first component as part of the foreground or background depending on (1) whether the component is the largest component on the image, or (2) whether the component has a largest or smallest depth relative to other individual components, or (3) both, wherein the first component is a first foreground component, and when a group of large components of similar size exists on the image, the large component with the smallest depth relative to the depth of the other large components is the first component; and
determining whether other components are classified as background or foreground based, as least in part, on individual distances from the other components to the first component;
wherein the area of the closest other component of the remaining other components is added to the first component and classified as part of the background or foreground when the distance is below a maximum threshold distance;
wherein the distance is a 3D Euclidean distance;
wherein the distance is measured from closest edge of the other component to closest edge of the first component;
wherein the area of the first component is re-established after the area of each component is added to the first component before determining the distances to the remaining other separate components; and
wherein the remaining other components are classified in the opposite segment of the first component when no remaining other component has a distance to the enlarged first component that is less than a maximum threshold distance.

17. A computer-implemented system of background-foreground segmentation for image processing, comprising:
at least one display;
at least one memory
at least one processor communicatively coupled to the display and the memory; and
a background-foreground segmentation unit operated by the at least one processor and to:
obtain pixel data comprising both non-depth data and depth data for at least one image, wherein the non-depth data comprises color data or luminance data or both and associated with the pixels;
determine whether a portion of the image is part of a background or foreground of the image based on the depth data and without using the non-depth data; and
determine whether a border area between the background and foreground formed by using the depth data are part of the background or foreground depending on the non-depth data without using the depth data.

18. The system of claim 17 wherein the border area is a thin band extending around at least a portion of the border between the depth based background and foreground.

19. The system of claim 18 wherein the border is about 10 pixels wide.

20. The system of claim 17 wherein non-depth data is only used to fill holes on the depth-based background and foregrounds and with the border area determination to determine the background and foreground segmentation.

21. The system of claim 17 wherein the segmentation of the border areas is determined by using at least one of an active contour deforming energy function algorithm and an iterative graph cutting algorithm.

22. The system of claim 17 wherein the background-foreground segmentation unit is arranged to:
designate a first component as part of the foreground or background depending on (1) whether the component is the largest component on the i

Assignees

• Intel Corp

Inventors

• Baruch Gilad
• Eldar Avigdor

Classifications

• G06T7/408

  Physics · mapped topic

• G06T7/0081 Primary

  Physics · mapped topic

• G06T7/0085

  Physics · mapped topic

• G06T2207/10024

  Color image · CPC title

• G06T2207/20148

  Physics · mapped topic