Method and apparatus for separating foreground image, and computer-readable recording medium

Patent metadata

Field	Value
Publication number	US-9600898-B2
Application number	US-201514918680-A
Country	US
Kind code	B2
Filing date	Oct 21, 2015
Priority date	Oct 23, 2014
Publication date	Mar 21, 2017
Grant date	Mar 21, 2017

Abstract

A method and an apparatus for separating a foreground image are disclosed. The method includes obtaining an input image and depth information of the input image; roughly dividing the input image to obtain a rough foreground region based on the depth information of the input image; obtaining motion information of the input image, and generating a three-color image from the rough foreground region based on the motion information; and separating the foreground image from the generated three-color image. According to the method, the three-color image is generated based on the depth information and the motion information, thus a foreground object can be accurately separated from the three-color image.

First claim

What is claimed is:

1. A method for separating a foreground image, the method comprising:
obtaining an input image and depth information of the input image;
roughly dividing, based on the depth information of the input image, the input image to obtain a rough foreground region;
obtaining motion information of the input image,
generating a three-color image from the input image based on the rough foreground region and the motion information;
first determining a relationship between the motion information of a plurality of pixels in the three-color image and a first threshold value;
second determining whether all pixels in an adjacent region of each of the plurality of pixels belong to the rough foreground region; and
separating the foreground image from the generated three-color image by identifying at least one first pixel of the plurality of pixels as part of the foreground image if the first determining determines that the motion information of the at least one first pixel is greater than or equal to the threshold and the second determining determines that all pixels in the adjacent region of the at least one first pixel belong to the rough foreground region.

2. The method for separating a foreground image according to claim 1 , wherein the generating the three-color image from the rough foreground region based on the motion information includes:
obtaining images of m continuous frames immediately before the input image and calculating the motion information of each pixel from the images of the m continuous frames;
third determining that at least one second pixel of the plurality of pixels belongs to a background region if the first determining determines that the motion information of the at least one second pixel is less than the first threshold; and
fourth determining that at least one third pixel of the plurality of pixels belongs to an unknown region, if the first determining determines that the motion information of the at least one third pixel is greater than or equal to the first threshold and the second determining determines that at least one in the adjacent region of the at least one third pixel does not belong to the rough foreground region.

3. The method for separating a foreground image according to claim 2 , wherein the separating the foreground image from the generated three-color image includes:
calculating color information, depth gradient information and depth change information of each of the at least one third pixel in the unknown region;
determining, based on the corresponding color information, the corresponding depth gradient information and the corresponding depth change information, whether each of the at least one third pixel belongs to the foreground region; and
acquiring, based on a determination result, the foreground image.

4. The method for separating a foreground image according to claim 3 , wherein the separating the foreground region from the generated three-color image includes:
constructing a cost function of an α value for separating the foreground image;
for each of the at least one third pixel in the unknown region, minimizing the corresponding cost function, and calculating the α value of each of the at least one third pixel by using the corresponding color information, the corresponding depth gradient information and the corresponding depth change information; and
determining that one or more of the at least one third pixel belongs to the foreground region when the α value the one or more of the at least one third pixel is greater than or equal to a second threshold, and
determining that the one or more of the at least one third pixel does not belong to the foreground region when the α value the one or more of the at least one third pixel is less than the second threshold.

5. The method for separating a foreground image according to claim 4 , further comprising:
for each of the at least one third pixel in the unknown region, calculating a normalized weight of the color information, the depth gradient information and the depth change information of at least one pixel in a region of n×n pixels adjacent to each of the at least one third pixel,
wherein the calculating calculates the α value by using a weighted value of the corresponding color information, a weighted value of the corresponding depth gradient information and a weighted value of the corresponding depth change information.

6. The method for separating a foreground image according to claim 1 , wherein the roughly dividing the input image to obtain the rough foreground region based on the depth information of the input image includes:
detecting, based on the depth information of the input image, a head-and-shoulders part of a person as the rough foreground region from the input image according to an Ω-shape feature; and
for each pixel other than the head-and-shoulders part in the input image, determining whether the pixel belongs to the rough foreground region by comparing depth value of the pixel and depth value of the detected head-and-shoulders part,
determining the pixel belongs to the rough foreground region when a difference between the depth value of the pixel and the depth value of the head-and-shoulders part is less than or equal to a third threshold, and
determining that the pixel does not belong to the rough foreground region when the difference between the depth value of the pixel and the depth value of the head-and-shoulders part is greater than the third threshold.

7. The method for separating a foreground image according to claim 1 , wherein the motion information is optical flow information.

8. An apparatus for separating a foreground image, the apparatus comprising:
a memory having computer-readable instructions stored therein; and
a processor configured to execute the computer-readable instructions to
obtain an input image and depth information of the input image,
roughly divide, based on the depth information of the input image, the input image to obtain a rough foreground region,
generate a three-color image from the input image based on the rough foreground region based and motion information of the input image,
first determine a relationship between the motion information of a plurality of pixels in the three-color image and a first threshold value,
second determine whether all pixels in an adjacent region of each of the plurality of pixels belong to the rough foreground region, and
separate the foreground image from the generated three-color image by identifying at least one first pixel of the plurality of pixels as part of the foreground image if the first determining determines that the motion information of the at least one first pixel is greater than or equal to the threshold and the second determining determines that all pixels in the adjacent region of the at least one first pixel belong to the rough foreground region.

9. The apparatus for separating a foreground image according to claim 8 , wherein the processor is configured to execute the computer-readable instructions to,
generate the three-color image by obtaining images of m continuous frames immediately before the input image and calculating the motion information of each pixel from the images of the m continuous frames,
third determine that at least one second pixel of the plurality of pixels belongs to a background region if the first determining determines that the motion information of the at least one second pixel is less than the first threshold; and
fourth determine that at least one third pixel of the plurality of pixels belongs to an unknown region, if the first determining determines that the motion information of the at least one third pixel is greater than or equal to the first threshold and the seco

Assignees

• Liu Liyan
• Zhao Ying
• Ricoh Co Ltd

Inventors

• Liu Liyan
• Zhao Ying

Classifications

• G06T2207/20144

  Physics · mapped topic

• G06T7/2006 Primary

  Physics · mapped topic

• G06T7/0097

  Physics · mapped topic

• G06T2207/10028

  Range image; Depth image; 3D point clouds · CPC title

• G06T7/174 Primary

  involving the use of two or more images · CPC title