Auto-classification system and method with dynamic user feedback

Patent metadata

Field	Value
Publication number	US-11238079-B2
Application number	US-201916272278-A
Country	US
Kind code	B2
Filing date	Feb 11, 2019
Priority date	Oct 31, 2012
Publication date	Feb 1, 2022
Grant date	Feb 1, 2022

Abstract

In an auto-classification system, example documents whose content exemplifies a content category or classification can be imported into a classification model. The classification model is tested to assess accuracy. Based on the testing, metrics or other information can be provided as feedback to a user. The user can iteratively refine the classification model and keep re-running the classifications to view how each change to the classification model improves accuracy. If no user refinement is desired, the auto-classification system classifies documents utilizing the classification model. This technology enhances the overall transparency and defensibility of the auto-classification process.

First claim

What is claimed is:

1. A computer-implemented method of automatic classification of digital content, the method comprising:
creating or modifying a classification model, the creating or modifying comprising importing, from a document source into the classification model, example documents having content that exemplifies a content category or classification such that the classification model comprises the example documents thus imported, the importing performed by an auto-classification system having a processor and a non-transitory computer-readable medium;
testing the classification model for accuracy assessment, the testing performed by the auto-classification system and comprising classifying test documents utilizing the classification model;
generating, by the auto-classification system based on the testing, feedback on the accuracy assessment of the classification model;
displaying, by the auto-classification system through a user interface on a user device, the feedback on the accuracy assessment of the classification model;
determining, by the auto-classification system based on an indication received through the user interface, whether to refine the classification model, the indication received through the user interface including an instruction to reject or accept the content category or classification of the example documents imported into the classification model;
responsive to the indication indicating user refinement of the classification model, iteratively performing the modifying, the testing, the generating, the displaying, and the determining;
classifying, by the auto-classification system utilizing the classification model, documents in a repository, the classifying comprising:
comparing a document in the repository relative to a set of example documents in the classification model; and
assigning the content category or classification of the set of example documents in the classification model to the document in the repository based on a similarity between the document in the repository and the set of example documents in the classification model.

2. The computer-implemented method according to claim 1 , wherein the classifying further comprises determining a classification confidence level for the document in the repository.

3. The computer-implemented method according to claim 1 , further comprising:
requesting, by the auto-classification system through the user interface, a user selection of a classification for importation of the example documents into the classification model.

4. The computer-implemented method according to claim 1 , further comprising:
requesting, by the auto-classification system through the user interface, a user decision on whether to allow sampling of the example documents as test documents.

5. The computer-implemented method according to claim 1 , further comprising:
automatically randomly selecting, by the auto-classification system, a plurality of example documents in the classification model as the test documents.

6. The computer-implemented method according to claim 1 , wherein the iteratively performing the modifying, the testing, the generating, the displaying, and the determining comprises:
for each iteration in which the classification model is tested for accuracy assessment and the feedback on the accuracy assessment of the classification model is updated based on the testing, updating, by the auto-classification system, the user interface to provide updated feedback based on latest iteration of the classification model.

7. The computer-implemented method according to claim 1 , further comprising:
determining, by the auto-classification system based on the feedback on the accuracy assessment of the classification model, a recommended action to refine the classification model; and
displaying, by the auto-classification system through the user interface on the user device, the recommended action to refine the classification model.

8. A computer program product for automatic classification of digital content, the computer program product comprising a non-transitory computer-readable medium storing instructions translatable by a processor of an auto-classification system for:
creating or modifying a classification model, the creating or modifying comprising importing, from a document source into the classification model, example documents having content that exemplifies a content category or classification such that the classification model comprises the example documents thus imported;
testing the classification model for accuracy assessment, the testing comprising classifying test documents utilizing the classification model;
generating, based on the testing, feedback on the accuracy assessment of the classification model;
displaying, through a user interface on a user device, the feedback on the accuracy assessment of the classification model;
determining, based on an indication received through the user interface, whether to refine the classification model, the indication received through the user interface including an instruction to reject or accept the content category or classification of the example documents imported into the classification model;
responsive to the indication indicating user refinement of the classification model, iteratively performing the modifying, the testing, the generating, the displaying, and the determining;
classifying, utilizing the classification model, documents in a repository, the classifying comprising:
comparing a document in the repository to a set of example documents in the classification model; and
assigning the content category or classification of the set of example documents in the classification model to the document in the repository.

9. The computer program product of claim 8 , wherein the classifying further comprises determining a classification confidence level for the document in the repository.

10. The computer program product of claim 8 , wherein the instructions are further translatable by the processor of the auto-classification system for:
requesting, through the user interface, a user selection of a classification for importation of the example documents into the classification model.

11. The computer program product of claim 8 , wherein the instructions are further translatable by the processor of the auto-classification system for:
requesting, through the user interface, a user decision on whether to allow sampling of the example documents as test documents.

12. The computer program product of claim 8 , wherein the instructions are further translatable by the processor of the auto-classification system for:
automatically randomly selecting a plurality of example documents in the classification model as the test documents.

13. The computer program product of claim 8 , wherein the iteratively performing the modifying, the testing, the generating, the displaying, and the determining comprises:
for each iteration in which the classification model is tested for accuracy assessment and the feedback on the accuracy assessment of the classification model is updated based on the testing, updating the user interface to provide updated feedback based on latest iteration of the classification model.

14. The computer program product of claim 8 , wherein the instructions are further translatable by the processor of the auto-classification system for:
determining, based on the feedback on the accuracy assessment of the classification model, a recommended action to refine the classification model; and
displaying, through the user interface on the user device, the recommended action to refine the cla

Assignees

• Open Text Corp

Inventors

• Simard Charles-Olivier
• Bowyer Alex
• Leclerc Daniel
• Molloy Steve

Classifications

• G06F16/285

  Clustering or classification · CPC title

• G06F3/04842

  Selection of displayed objects or displayed text elements (G06F3/0482 takes precedence) · CPC title

• G06F16/35 Primary

  Clustering; Classification · CPC title

• G06F16/93

  Document management systems · CPC title

• G06F16/258

  Data format conversion from or to a database · CPC title