Auto-classification system and method with dynamic user feedback

Patent metadata

Field	Value
Publication number	US-11720618-B2
Application number	US-202217584179-A
Country	US
Kind code	B2
Filing date	Jan 25, 2022
Priority date	Oct 31, 2012
Publication date	Aug 8, 2023
Grant date	Aug 8, 2023

Abstract

In an auto-classification system, example documents whose content exemplifies a content category or classification can be imported into a classification model. The classification model is tested to assess accuracy. Based on the testing, metrics or other information can be provided as feedback to a user. The user can iteratively refine the classification model and keep re-running the classifications to view how each change to the classification model improves accuracy. If no user refinement is desired, the auto-classification system classifies documents utilizing the classification model. This technology enhances the overall transparency and defensibility of the auto-classification process.

First claim

What is claimed is:

1. A method, comprising:
displaying, by a computer through a user interface on a user device, a plurality of classifications for example documents contained in a classification model, the displaying including displaying each of the plurality of classifications with a user-selectable interface element;
determining, by the computer, one or more classifications for the example documents contained in the classification model, whether the determining comprises determining whether to accept or reject each of the plurality of classifications based on user input via the user-selectable interface element displayed with each of the plurality of classifications; and
classifying, by the computer utilizing the classification model, documents in a repository, the classifying comprising:
comparing a document in the repository to a set of example documents in the classification model; and
assigning a content category or classification of the set of example documents in the classification model to the document in the repository.

2. The method according to claim 1 , wherein the displaying is automatically activated without user intervention or user input.

3. The method according to claim 1 , wherein the user-selectable interface element comprises an accept button and a reject button.

4. The method according to claim 1 , further comprising:
identifying a problem or deficiency of the classification model;
determining one or more remedial actions; and
presenting, through the user interface, the one or more remedial actions for refining the classification model.

5. The method according to claim 4 , wherein the problem or deficiency relates to an overall accuracy of the classification model.

6. The method according to claim 1 , further comprising:
determining recommendations, recommended actions, suggestions, tips, guiding information, or a combination thereof for refining the classification model; and
presenting the recommendations, the recommended actions, the suggestions, the tips, the guiding information, or the combination thereof on the user interface.

7. The method according to claim 6 , further comprising:
prior to the presenting, prioritizing the recommendations, the recommended actions, the suggestions, the tips, the guiding information, or the combination thereof.

8. A system, comprising:
a processor;
a non-transitory computer-readable medium; and
instructions stored on the non-transitory computer-readable medium and translatable by the processor for:
displaying, through a user interface on a user device, a plurality of classifications for example documents contained in a classification model, the displaying including displaying each of the plurality of classifications with a user-selectable interface element;
determining one or more classifications for the example documents contained in the classification model, whether the determining comprises determining whether to accept or reject each of the plurality of classifications based on user input via the user-selectable interface element displayed with each of the plurality of classifications; and
classifying, utilizing the classification model, documents in a repository, the classifying comprising:
comparing a document in the repository to a set of example documents in the classification model; and
assigning a content category or classification of the set of example documents in the classification model to the document in the repository.

9. The system of claim 8 , wherein the displaying is automatically activated without user intervention or user input.

10. The system of claim 8 , wherein the user-selectable interface element comprises an accept button and a reject button.

11. The system of claim 8 , wherein the instructions are further translatable by the processor for:
identifying a problem or deficiency of the classification model;
determining one or more remedial actions; and
presenting, through the user interface, the one or more remedial actions for refining the classification model.

12. The system of claim 11 , wherein the problem or deficiency relates to an overall accuracy of the classification model.

13. The system of claim 8 , wherein the instructions are further translatable by the processor for:
determining recommendations, recommended actions, suggestions, tips, guiding information, or a combination thereof for refining the classification model; and
presenting the recommendations, the recommended actions, the suggestions, the tips, the guiding information, or the combination thereof on the user interface.

14. The system of claim 13 , wherein the instructions are further translatable by the processor for:
prior to the presenting, prioritizing the recommendations, the recommended actions, the suggestions, the tips, the guiding information, or the combination thereof.

15. A computer program product comprising a non-transitory computer-readable medium storing instructions translatable by a processor for:
displaying, through a user interface on a user device, a plurality of classifications for example documents contained in a classification model, the displaying including displaying each of the plurality of classifications with a user-selectable interface element;
determining one or more classifications for the example documents contained in the classification model, whether the determining comprises determining whether to accept or reject each of the plurality of classifications based on user input via the user-selectable interface element displayed with each of the plurality of classifications; and
classifying, utilizing the classification model, documents in a repository, the classifying comprising:
comparing a document in the repository to a set of example documents in the classification model; and
assigning a content category or classification of the set of example documents in the classification model to the document in the repository.

16. The computer program product of claim 15 , wherein the displaying is automatically activated without user intervention or user input.

17. The computer program product of claim 15 , wherein the user-selectable interface element comprises an accept button and a reject button.

18. The computer program product of claim 15 , wherein the instructions are further translatable by the processor for:
identifying a problem or deficiency of the classification model;
determining one or more remedial actions; and
presenting, through the user interface, the one or more remedial actions for refining the classification model, wherein the problem or deficiency relates to an overall accuracy of the classification model.

19. The computer program product of claim 15 , wherein the instructions are further translatable by the processor for:
determining recommendations, recommended actions, suggestions, tips, guiding information, or a combination thereof for refining the classification model; and
presenting the recommendations, the recommended actions, the suggestions, the tips, the guiding information, or the combination thereof on the user interface.

20. The computer program product of claim 19 , wherein the instructions are further translatable by the processor for:
prior to the presenting, prioritizing the recommendations, the recommended actions, the suggestions, the tips, the guiding information, or the combination thereof.

Assignees

• Open Text Corp

Inventors

• Simard Charles-Olivier
• Bowyer Alex
• Leclerc Daniel
• Molloy Steve

Classifications

• G06F16/35 Primary

  Clustering; Classification · CPC title

• G06F3/0482

  Interaction with lists of selectable items, e.g. menus · CPC title

• G06F3/04842

  Selection of displayed objects or displayed text elements (G06F3/0482 takes precedence) · CPC title

• G06F16/258

  Data format conversion from or to a database · CPC title

• G06F16/285

  Clustering or classification · CPC title