Method and apparatus for classifying pixels in an input image and image processing system

Patent metadata

Field	Value
Publication number	US-9576201-B2
Application number	US-201414338127-A
Country	US
Kind code	B2
Filing date	Jul 22, 2014
Priority date	Jul 25, 2013
Publication date	Feb 21, 2017
Grant date	Feb 21, 2017

Abstract

The present disclosure relates to a method and apparatus for classifying pixels in an input image and image processing system. The apparatus includes a first confidence determination unit configured for determining a first confidence of each of the pixels in the input image belonging to each of a plurality of predetermined classes; a detection unit configured for acquiring, by using a detector for detecting a particular class among the plurality of predetermined classes, detection results used to indicate pixels which belong to the particular class; an updating unit configured for updating the first confidences according to the detection results; and a class determination unit configured for determining the class to which each of the pixels in the input image belongs among the plurality of predetermined classes, according to the updated first confidences.

First claim

What is claimed is:

1. An apparatus to classify pixels in an input image, the apparatus comprising:
at least one processing unit coupled to a memory;
a first confidence determination unit configured to determine a first confidence of each pixel in the pixels of the input image that represents a probability that a given pixel belongs to a particular class, wherein each pixel belongs to each of a plurality of predetermined classes;
a detection unit configured to acquire, by using a detector to detect a particular class for a pixel among the plurality of predetermined classes, detection results used as an indication of the particular class to which each pixel in the input image belongs among the plurality of predetermined classes;
an updating unit configured to update the first confidences according to the acquired detection results of the detector; and
a class determination unit configured to determine, according to the updated first confidences, the class to which each pixel in the input image belongs among the plurality of predetermined classes,
wherein at least one of the first confidence determination unit, the detection unit, the updating unit, and the class determination unit are implemented by the at least one processing unit.

2. The apparatus according to claim 1 , wherein the updating unit includes:
an update pixel determination sub-unit configured to determine a plurality of pixels whose first confidences need to be updated, according to the detection results, and
a first updating sub-unit configured to update, for each of the plurality of pixels, the first confidence of the pixel according to a probability of this pixel belonging to a foreground of the input image.

3. The apparatus according to claim 2 ,
wherein super-pixels are obtained by segmenting the input image and a confidence of a pixel is determined according to a confidence of a super-pixel where the pixel is located,
wherein the update pixel determination sub-unit further is configured to determine pixels included in the super-pixels covered by a region formed of the pixels which belong to the particular class as the pixels whose first confidences need to be updated, and
wherein the first updating sub-unit includes:
a second confidence determination sub-unit configured to determine, for each of the super-pixels covered by the region formed of the pixels which belong to the particular class, a second confidence of the super-pixel belonging to each of the plurality of predetermined classes, according to the probabilities of the pixels included in the super-pixel belonging to the foreground of the input image, and
an updating confidence sub-unit configured to update, for each of the super-pixels covered by the region formed of the pixels which belong to the particular class, the first confidences of the pixels included in the super-pixel according to the second confidence of the super-pixel.

4. The apparatus according to claim 3 ,
wherein, for each of the super-pixels covered by the region formed of the pixels which belong to the particular class, the second confidence of the super-pixel belonging to the particular class is equal to an average value of the probability of each pixel included in the super-pixel belonging to the foreground of the input image, and the second confidence of the super-pixel belonging to each of other predetermined classes is equal to an average value of a probability of each pixel included in the super-pixel belonging to the corresponding one of the other predetermined classes, and
wherein the probability of the pixel belonging to each of the other predetermined classes is equal to (1−the probability of the pixel belonging to the foreground of the input image)/(the number of the plurality of predetermined classes−1).

5. The apparatus according to claim 3 , wherein the updating confidence sub-unit further is configured to update, for each of the pixels included in the super-pixel, the first confidence of the pixel, by linearly combining the second confidence of the super-pixel with the first confidence of the pixel.

6. The apparatus according to claim 5 , wherein a weight assigned to the second confidence is larger than a weight assigned to the first confidence.

7. The apparatus according to claim 3 , wherein the super-pixels covered by the region formed of the pixels which belong to the particular class include a super-pixel which has a ratio larger than a first predetermined threshold, wherein the ratio is equal to the number of this super-pixel's pixels overlapped with the region formed of the pixels which belong to the particular class divided by the total number of this super-pixel's pixels.

8. The apparatus according to claim 1 , wherein the updating unit includes:
an update region acquisition sub-unit configured to determine, within a region formed of the pixels which belong to the particular class, an update region according to the detection result, and
a second updating sub-unit configured to classify the update region as a whole using a classifier, and to update the first confidence of each pixel in the update region according to an output of the classifier.

9. The apparatus according to claim 8 , wherein the classifier is trained by a training unit, wherein the training unit includes:
an over-segmentation sub-unit configured to over segment a training image into a set of regions,
a grouping sub-unit configured to group the regions which belong to the particular class, to obtain a grouping region, and
a generation sub-unit configured to generate the classifier according to features of the grouping region and other un-grouped regions.

10. The apparatus according to claim 9 ,
wherein super-pixels are obtained by segmenting the input image and a confidence of a pixel is determined according to a confidence of a super-pixel where the pixel is located, and,
wherein the second updating sub-unit includes:
a grouping sub-unit configured to obtain a grouping super-pixel by grouping the super-pixels covered by the update region among a plurality of super-pixels into which the input image is segmented, and
an updating grouping sub-unit configured to update the first confidences of the pixels included in each super-pixel forming the grouping super-pixel, according to a confidence of the grouping super-pixel determined by using the classifier.

11. The apparatus according to claim 10 , wherein the super-pixels covered by the update region include a super-pixel which has a ratio of the number of its pixels overlapped with the update region to the total number of its pixels larger than a second predetermined threshold.

12. The apparatus according to claim 1 , wherein the detection unit includes:
a first detection sub-unit configured to detect a plurality of image regions including the pixels which belong to the particular class in the input image by using the detector, and
a first determination sub-unit configured to determine a region generated by grouping the plurality of image regions as the detecting results.

13. The apparatus according to claim 1 , wherein the detection unit includes:
a second detection sub-unit configured to detect, in a case that the particular class is a human class, a face region in the input image by using the detector being a face detector, and
a second determination sub-unit configured to determine a region generated by enlarging the face region as the detection results.

14. An image processing system, comprising:
the apparatus according to claim 1 .

15. A method for an apparatus to classify pixels in an input image, the method comprising:
determining a first confi

Assignees

• Canon Kk

Inventors

• Wu Bo
• Jiang Yong
• Lv you
• Xu Xingwei

Classifications

• G06V10/809

  of classification results, e.g. where the classifiers operate on the same input data · CPC title

• G06V10/764

  using classification, e.g. of video objects · CPC title

• G06F18/2431

  Multiple classes · CPC title

• G06F18/254

  of classification results, e.g. of results related to same input data · CPC title

• G06V20/10 Primary

  Terrestrial scenes (scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59) · CPC title