Auto-classification system and method with dynamic user feedback

Patent metadata

Field	Value
Publication number	US-9348899-B2
Application number	US-201213665607-A
Country	US
Kind code	B2
Filing date	Oct 31, 2012
Priority date	Oct 31, 2012
Publication date	May 24, 2016
Grant date	May 24, 2016

Abstract

An auto-classification system and method provides dynamic user feedback in a guide that is presented to the user. The feedback presented in the guide enables the user to refine the classification model by adding or removing exemplars, creating, editing or deleting rules, or performing other such adjustments to the classification model. This technology enhances the overall transparency and defensibility of the auto-classification process.

First claim

The invention claimed is:

1. A computer-implemented method of automatically classifying digital content, the method comprising:
creating, by an auto-classification system embodied on non-transitory computer memory, a classification model for classifying digital content, wherein creating the classification model comprises:
prompting a user via a user interface to enter a name and description for the classification model, and
responsive to user selection of one or more documents via the user interface, adding or importing, from a document source into a container, the selected one or more documents as exemplars for a category of digital content within the classification model;

subsequent to a user selecting a plurality of documents, the auto-classification system classifying the plurality of documents using characteristics of the exemplars;
during the classifying, after the classifying is complete, or both during the classifying and after the classifying is complete,
the auto-classification system determining performance metrics representing accuracy of the classification model in classifying the plurality of documents, the performance metrics including precision, recall, match, noise, and silence metrics;
the auto-classification system determining one or more recommended actions based on the performance metrics;
the auto-classification system displaying the performance metrics including the precision, recall, match, noise, and silence metrics for a number of the plurality of documents that have been classified using the classification model;
the auto-classification system displaying a user feedback guide that presents the one or more recommended actions to improve the accuracy of the classification model, wherein the one or more recommended actions comprise adding one or more exemplars from the number of the plurality of documents that have been classified using the classification model or specifying how to meet one or more associated content classification rules; and
providing a user interface element associated with the user feedback guide that, in response to receipt of user input, causes a recommended action to be performed on the classification model.

2. The method as claimed in claim 1 wherein displaying the user feedback guide comprises presenting a recommended action to change a number of exemplars.

3. The method as claimed in claim 1 wherein displaying the user feedback guide comprises presenting a recommended action to review a greater number of classified documents to improve a confidence level.

4. The method as claimed in claim 1 wherein displaying the performance metrics comprises displaying values for precision and recall, wherein the precision represents a frequency with which an assigned classification matches an expected classification and wherein the recall represents a frequency with which the expected classification is assigned across all processed documents.

5. The method as claimed in claim 1 wherein the performance metrics are displayed in response to a review of a collection of classified documents, wherein the review comprises accepting or rejecting an automatically assigned classification and wherein the review further comprises specifying an expected classification for any misclassified documents.

6. The method as claimed in claim 1 wherein creating the classification model further comprises defining at least one classification rule, wherein the rule comprises a rule priority determining an order in which rule is applied, a confidence level to be applied to a document when the document satisfies a condition specified by the rule and an applied classification that is to be applied to the document.

7. The method as claimed in claim 1 comprising prioritizing the recommended actions based on importance and displaying the recommended actions in order of priority.

8. A non-transitory computer-readable medium comprising programmed instructions in code which, when loaded into a memory and executed by a processor of a computing device, causes the computing device to:
create a classification model for classifying digital content, wherein creating the classification model comprises:
prompting a user via a user interface to enter a name and description for the classification model, and
responsive to user selection of one or more documents via the user interface, adding or importing, from a document source into a container, the selected one or more documents as exemplars for a category of digital content within the classification model;

subsequent to a user selecting a plurality of documents, the auto-classification system classify the plurality of documents using characteristics of the exemplars;
during the classifying, after the classifying is complete, or both during the classifying and after the classifying is complete,
determine performance metrics representing accuracy of the classification model in classifying the plurality of documents, the performance metrics including precision, recall, match, noise, and silence metrics;
determine one or more recommended actions based on the performance metrics;
display the performance metrics including the precision, recall, match, noise, and silence metrics for a number of the plurality of documents that have been classified using the classification model;
display a user feedback guide that presents the one or more recommended actions to improve the accuracy of the classification model, wherein the one or more recommended actions comprise adding one or more exemplars from the number of the plurality of documents that have been classified using the classification model or specifying how to meet one or more associated content classification rules; and
provide a user interface element associated with the user feedback guide that, in response to receipt of user input, causes a recommended action to be performed on the classification model.

9. The non-transitory computer-readable medium as claimed in claim 8 comprising code to present a recommended action to change a number of exemplars.

10. The non-transitory computer-readable medium as claimed in claim 8 comprising code to present a recommended action to review a greater number of classified documents to improve a confidence level.

11. The non-transitory computer-readable medium as claimed in claim 8 comprising code to display values for precision and recall, wherein the precision represents a frequency with which an assigned classification matches an expected classification and wherein the recall represents a frequency with which the expected classification is assigned across all processed documents.

12. The non-transitory computer-readable medium as claimed in claim 8 wherein the performance metrics are displayed in response to a review of a collection of classified documents, wherein the review comprises accepting or rejecting an automatically assigned classification and wherein the review further comprises specifying an expected classification for any misclassified documents.

13. The non-transitory computer-readable medium as claimed in claim 8 comprising code to define at least one classification rule, wherein the rule comprises a rule priority determining an order in which rule is applied, a confidence level to be applied to a document when the document satisfies a condition specified by the rule and an applied classification that is to be applied to the document.

14. The non-transitory computer-readable medium as claimed in claim 8 comprising code for prioritizing the recommended actions based on importance and for displaying the recommended actions in order of priority.

15. An auto

Assignees

• Open Text Corp

Inventors

• Simard Charles-Olivier
• Bowyer Alex
• Leclerc Daniel
• Molloy Steve

Classifications

• G06F16/35 Primary

  Clustering; Classification · CPC title

• G06F16/93

  Document management systems · CPC title

• G06F16/258

  Data format conversion from or to a database · CPC title

• G06F16/285

  Clustering or classification · CPC title

• G06F3/04842

  Selection of displayed objects or displayed text elements (G06F3/0482 takes precedence) · CPC title