Score fusion and training data recycling for video classification

Patent metadata

Field	Value
Publication number	US-9147129-B2
Application number	US-201213622328-A
Country	US
Kind code	B2
Filing date	Sep 18, 2012
Priority date	Nov 18, 2011
Publication date	Sep 29, 2015
Grant date	Sep 29, 2015

Abstract

Multiple classifiers can be applied independently to evaluate images or video. Where there are heavily imbalanced class distributions, a local expert forest model for meta-level score fusion for event detection can be used. Performance variations of classifiers in different regions of a score space can be adapted. Multiple pairs of experts based on different partitions, or “trees,” can form a “forest,” balancing local adaptivity and over-fitting. Among ensemble learning methods, stacking with a meta-level classifier can be used to fuse an output of multiple base-level classifiers to generate a final score. A knowledge-transfer framework can reutilize the base-training data for learning the meta-level classifier. By recycling the knowledge obtained during a base-classifier-training stage, efficient use can be made of all available information, such as can be used to achieve better fusion and better overall performance.

First claim

The claimed invention is:

1. A system, comprising:
a processor circuit, including:
a first data input configured to receive probability estimates from two or more separate feature classifiers over a collection of training items, those items having associated ground truth category information; and

a processor-readable medium, including instructions that, when performed by the processor, configure the system to:
select a fusion model to adapt local statistics of the two or more separate feature classifiers over the collection of training items;
generate K partitions on each separate feature classifier to form K 2 pairs of associations;
determine a maximum likelihood estimate of a pair of the K 2 pairs being the correct classifier including modelling the likelihood using a localized expert forest and using a linear model for the localized expert forest; and
fuse the maximum likelihood estimates from the separate feature classifiers according to the selected fusion model to generate an output probability estimate for new items that do not have associated ground truth information.

2. The system of claim 1 , wherein the processor-readable medium includes instructions that, when performed by the processor, configure the system to fuse the probability estimates from the separate feature classifiers using weights assigned to each classifier.

3. The system of claim 2 , wherein the processor-readable medium includes instructions that, when performed by the processor, configure the system to use an objective function to fuse the probability estimates from the separate feature classifiers, the objective function comprising a minimum mean squared error fusion function with a non-negative constraint.

4. The system of claim 2 , wherein the processor-readable medium includes instructions that, when performed by the processor, configure the system to use an objective function to fuse the probability estimates from the separate feature classifiers, the objective function comprising a linear support vector machine with a non-negative constraint.

5. The system of claim 1 , wherein the first data input is configured to receive probability estimates from the two or more separate feature classifiers over a collection of training items, wherein the collection of training items comprises video clips or still images that can be categorized by an activity depicted in the video clips or still images.

6. The system of claim 1 , wherein the processor-readable medium includes instructions that, when performed by the processor, configure the system to compare the generated output probability estimate to a threshold to identify a category for new items that do not have associated ground truth information.

7. The system of claim 1 , further comprising instructions that, when performed by the processor, configure the system to select K associations of the K 2 associations with the highest determined maximum likelihood scores.

8. The system of claim 7 , further comprising instructions that, when performed by the processor, configure the system to determine a cluster center for each of the K selected associations.

9. The system of claim 8 , further comprising instructions that, when performed by the processor, configure the system to perform linear discriminant analysis to determine a one dimensional projection vector that separates pairs of cluster centers.

10. The system of claim 9 , wherein instructions for determining a one dimensional projection vector that separates a pair of cluster centers, include instructions that, when performed by the processor, configure the system to project the determined cluster centers onto a one dimensional axis corresponding to the one dimensional projection vector.

11. The system of claim 10 , wherein instructions for determining a one dimensional projection vector that separates a pair of cluster centers include instructions that, when performed by the processor, configure the system to partition the one dimensional axis into partitions based on a specified threshold.

12. A method, comprising:
receiving probability estimates from two or more separate feature classifiers over a collection of training items, those items having associated ground truth category information;
selecting a fusion model to adapt to the local statistics of the separate feature classifiers over the training data;
generating K partitions on each separate feature classifier to form K 2 pairs of associations;
determining a maximum likelihood estimate of a pair of the K 2 pairs being the correct classifier including modelling the likelihood using a localized expert forest and using a linear model for the localized expert forest; and
fusing the maximum likelihood estimates from the separate feature classifiers according to the model to generate an output probability estimate for new items without associated ground truth information.

13. The method of claim 12 wherein the fusion model comprises weights assigned to each classifier.

14. The method of claim 13 wherein an objective function for fusing the local statistics comprises a minimum mean squared error fusion with non-negative constraint.

15. The method of claim 13 wherein an objective function for fusing the local statistics comprises a linear support vector machine with non-negative constraint.

16. The method of claim 12 , wherein the items are video clips and the categories denote activities depicted by the video clip.

17. The method of claim 12 , wherein fusing the probability estimates to generate an output probability estimate includes using the output probability estimate to identify a category for new items that do not have associated ground truth information.

Assignees

• Honeywell Int Inc

Inventors

• Liu Jingchen
• Mccloskey Scott

Classifications

• G06V10/7788

  the supervisor being a human, e.g. interactive learning with a human teacher · CPC title

• G06V10/774

  Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting · CPC title

• G06V10/809

  of classification results, e.g. where the classifiers operate on the same input data · CPC title

• G06F18/214

  Generating training patterns; Bootstrap methods, e.g. bagging or boosting · CPC title

• G06N20/00 Primary

  Machine learning · CPC title