Method and apparatus for separating foreground image, and non-transitory computer-readable recording medium

Patent metadata

Field	Value
Publication number	US-9767567-B2
Application number	US-201514930009-A
Country	US
Kind code	B2
Filing date	Nov 2, 2015
Priority date	Nov 12, 2014
Publication date	Sep 19, 2017
Grant date	Sep 19, 2017

Abstract

A method and an apparatus for separating a foreground image are disclosed. The method includes obtaining an input image, and color information and depth information of the input image; roughly dividing, based on the depth information of the input image, the input image to obtain an initial three-color image; reducing or expanding, based on the color information of the input image, an unknown region in the initial three-color image to obtain an optimized three-color image; and separating the foreground image from the optimized three-color image. According to the method, the initial three-color image can be optimized based on the color information of the input image, so that a more accurate three-color image can be obtained; thus the foreground image can be accurately separated from the three-color image.

First claim

What is claimed is:

1. A method for separating a foreground image, the method comprising:
obtaining an input image, and color information and depth information of the input image;
roughly dividing, based on the depth information of the input image, the input image to obtain an initial three-color image, the initial three-color image including a foreground image, a background image, and an unknown region;
reducing or expanding, based on the color information of the input image, the unknown region in the initial three-color image to obtain an optimized three-color image, the reducing or expanding including moving a boundary of the unknown region towards or away from a boundary of the foreground image;
separating the foreground image from the optimized three-color image; and
wherein the reducing or expanding the unknown region in the initial three-color image based on the color information of the input image further includes,
dividing the unknown region of the initial three-color image into a plurality of local regions, and
for each local region, determining whether color distribution in the local region satisfies a desired condition, and reducing or expanding the local region until the color distribution of the local region satisfies the desired condition when the desired condition is not satisfied, wherein the desired condition is a condition in which there are two kinds of color in the local region.

2. The method according to claim 1 , further comprising:
detecting a gap region between constituent parts of a foreground object; and
marking the gap region in a reduced or expanded three-color image of the initial three-color image as a part of the unknown region, so as to obtain the optimized three-color image.

3. The method according to claim 1 ,
wherein the separating the foreground image from the optimized three-color image comprises:
for each pixel in the unknown region of the optimized three-color image, selecting an optimum pair of a foreground pixel and a background pixel of the pixel from an adjacent region of the pixel;
estimating, based on color values of the optimum pair of the foreground pixel and the background pixel, a foreground component and a background component of the pixel;
calculating, based on the estimated foreground component and background component of the pixel, an Alpha-value of the pixel; and
separating, based on a foreground region in the optimized three-color image and the Alpha-values of the pixels in the unknown region, the foreground image.

4. The method according to claim 3 ,
wherein the selecting the optimum pair of the foreground pixel and the background pixel of the pixel from the adjacent region of the pixel comprises:
performing a weighted linear transformation for the color information of the input image to obtain the transformed color information of the input image, by using the depth information of the input image as a weight;
for each pixel in the unknown region in the optimized three-color image, selecting a plurality of foreground pixel samples from the foreground region in the adjacent region of the pixel, and selecting a plurality of background pixel samples from a background region in the adjacent region of the pixel, so as to constitute a sample set;
for pairs of foreground pixels and background pixels constituted of any foreground pixel and any background pixel in the sample set, constructing, based on the transformed color information and spatial position information of the pairs of the foreground pixels and the background pixels, an energy function; and
obtaining a pair of the foreground pixel and the background pixel that minimize the energy function, as the optimum pair of the foreground pixel and the background pixel.

5. The method according to claim 4 ,
wherein the estimating the foreground component and the background component of the pixel based on the color values of the optimum pair of the foreground pixel and the background pixel comprises:
for each pixel in the adjacent region of the pixel, obtaining the corresponding optimum pair of the foreground pixel and the background pixel;
calculating an average value of the color value of the foreground pixel in the optimum pair of the foreground pixel and the background pixel of the pixel and color values of the foreground pixels in the optimum pairs of the foreground pixels and the background pixels of pixels in the adjacent region of the pixel, as an estimating value of the foreground component of the pixel; and
calculating an average value of the color value of the background pixel in the optimum pair of the foreground pixel and the background pixel of the pixel and color values of the background pixels in the optimum pairs of the foreground pixels and the background pixels of pixels in the adjacent region of the pixel, as an estimating value of the background component of the pixel.

6. The method according to claim 5 ,
wherein the calculating the Alpha-value of the pixel based on the estimated foreground component and background component of the pixel comprises:
calculating an Alpha-estimating-value using the estimated foreground component and background component of the pixel;
calculating offsets between the estimated foreground component and background component, and an actual color value of the pixel;
calculating, based on the Alpha-estimating-value of the pixel and the offsets, a confidence level for calculating the Alpha-value of the pixel using the estimated foreground component and background component of the pixel; and
calculating, based on the Alpha-estimating-value and the confidence level, the Alpha-value of the pixel.

7. An apparatus for separating a foreground image, the apparatus comprising:
a memory having computer readable instructions stored thereon; and
at least one processor configured to execute the computer readable instructions to,
obtain an input image, and color information and depth information of the input image;
roughly divide, based on the depth information of the input image, the input image to obtain an initial three-color image, the initial three-color image including a foreground image, a background image, and an unknown region;
reduce or expand, based on the color information of the input image, the unknown region in the initial three-color image to obtain an optimized three-color image, the reducing or expanding including moving a boundary of the unknown region towards or away from a boundary of the foreground image;
separate the foreground image from the optimized three-color image;
divide the unknown region of the initial three-color image into a plurality of local regions; and
for each local region, determine whether color distribution in the local region satisfies a desired condition, and reduce or expand the local region until the color distribution of the local region satisfies the desired condition when the desired condition is not satisfied, wherein the desired condition is a condition in which there are two kinds of color in the local region.

8. The apparatus according to claim 7 , wherein the at least one processor is further configured to:
detect a gap region between constituent parts of a foreground object; and
mark the gap region in a reduced or expanded three-color image of the initial three-color image as a part of the unknown region, so as to obtain the optimized three-color image.

9. The apparatus according to claim 7 , wherein the at least one processor is further configured to separate the foreground image from the optimized three-color image by:
for each pixel in the unknown region of the optimized three-color image, selecting an optimum pair of a foreground pixel and a background pixel of the pixel from an adjacent region of

Assignees

• Zhao Ying
• Wang Gang
• Liu Liyan
• Ricoh Co Ltd

Inventors

• Zhao Ying
• Wang Gang
• Liu Liyan

Classifications

• G06T7/194

  involving foreground-background segmentation · CPC title

• G06T2207/10028

  Range image; Depth image; 3D point clouds · CPC title

• G06T7/11 Primary

  Region-based segmentation · CPC title

• G06T2207/10004

  Still image; Photographic image · CPC title

• G06V10/25

  Determination of region of interest [ROI] or a volume of interest [VOI] · CPC title