Auto-classification system and method with dynamic user feedback

Patent metadata

Field	Value
Publication number	US-12332932-B2
Application number	US-202318330461-A
Country	US
Kind code	B2
Filing date	Jun 7, 2023
Priority date	Oct 31, 2012
Publication date	Jun 17, 2025
Grant date	Jun 17, 2025

Abstract

In an auto-classification system, example documents whose content exemplifies a content category or classification can be imported into a classification model. The classification model is tested to assess accuracy. Based on the testing, metrics or other information can be provided as feedback to a user. The user can iteratively refine the classification model and keep re-running the classifications to view how each change to the classification model improves accuracy. If no user refinement is desired, the auto-classification system classifies documents utilizing the classification model. This technology enhances the overall transparency and defensibility of the auto-classification process.

First claim

What is claimed is:

1. A method, comprising:
receiving, by a processor through a user interface on a user device, a plurality of classifications for example documents contained in a classification model, each respective classification of the plurality of classifications is displayed in the user interface with associated user-selectable interface elements that enable the user device to accept or reject the respective classification based on an input from the user interface;
determining, by the processor, one or more classifications for the example documents contained in the classification model based on one or more selections of the user- selectable interface elements; and
classifying, by the processor utilizing the example documents contained in the classification model, a set of documents based on the one or more classifications.

2. The method according to claim 1 , further comprising:
comparing each respective document in the set of documents to a set of example documents contained in the classification model;
assigning a content category or classification of the set of example documents contained in the classification model to the respective document in the set of documents;
during the classifying, after the classifying is complete, or both during the classifying and after the classifying is complete, determining a level of certainty that the content category or classification is correctly assigned to the respective document.

3. The method according to claim 1 , wherein the user interface comprises a dynamic user feedback guide for optimizing accuracy of the classification model through remedial actions, the remedial actions including adding more documents to the set of documents, adding more example documents to the set of example documents, or both.

4. The method according to claim 3 , wherein the dynamic user feedback guide is automatically activated and displayed without user intervention or user input.

5. The method according to claim 3 , wherein the dynamic user feedback guide is activated and displayed only when the accuracy of the classification model falls below a predetermined threshold.

6. The method according to claim 3 , wherein the dynamic user feedback guide is only activated and displayed in response to user input on a user-selectable interface element.

7. The method according to claim 1 , wherein the user interface comprises a review pane, wherein the plurality of classifications and the associated user-selectable interface elements are displayed in the review pane.

8. A system, comprising:
a processor;
a non-transitory computer-readable medium, operatively coupled with the processor; and
instructions stored on the non-transitory computer-readable medium that, when executed by the processor, cause the processor to:
receive, through a user interface on a user device, a plurality of classifications for example documents contained in a classification model, each respective classification of the plurality of classifications is displayed in the user interface with associated user-selectable interface elements that enable the user device to accept or reject the respective classification based on an input from the user interface;
determine one or more classifications for the example documents contained in the classification model based on one or more selections of the user-selectable interface elements; and
classify, utilizing the example documents contained in the classification model, a set of documents based on the one or more classifications.

9. The system of claim 8 , wherein the processor is further to:
compare each respective document in the set of documents to a set of example documents contained in the classification model;
assign a content category or classification of the set of example documents contained in the classification model to the respective document in the set of documents;
during the classification, after the classifying is complete, or both during the classification and after the classification is complete, determine a level of certainty that the content category or classification is correctly assigned to the respective document.

10. The system of claim 8 , wherein the user interface comprises a dynamic user feedback guide for optimizing accuracy of the classification model through remedial actions, the remedial actions including adding more documents to the set of documents, adding more example documents to the set of example documents, or both.

11. The system of claim 10 , wherein the dynamic user feedback guide is automatically activated and displayed without user intervention or user input.

12. The system of claim 10 , wherein the dynamic user feedback guide is activated and displayed only when the accuracy of the classification model falls below a predetermined threshold.

13. The system of claim 10 , wherein the dynamic user feedback guide is only activated and displayed in response to user input on a user-selectable interface element.

14. The system of claim 8 , wherein the user interface comprises a review pane, wherein the plurality of classifications and the associated user-selectable interface elements are displayed in the review pane.

15. A computer program product comprising a non-transitory computer-readable medium storing instructions that, when executed by a processor, cause the processor to:
receive, through a user interface on a user device, a plurality of classifications for example documents contained in a classification model, each respective classification of the plurality of classifications is displayed in the user interface with associated user-selectable interface elements that enable the user device to accept or reject the respective classification based on an input from the user interface;
determine, by the processor, one or more classifications for the example documents contained in the classification model based on one or more selections of the user- selectable interface elements; and
classify, utilizing the example documents contained in the classification model, a set of documents based on the one or more classifications.

16. The computer program product of claim 15 , wherein the processor is further to:
compare each respective document in the set of documents to a set of example documents contained in the classification model;
assign a content category or classification of the set of example documents contained in the classification model to the respective document in the set of documents;
during the classification, after the classifying is complete, or both during the classification and after the classification is complete, determine a level of certainty that the content category or classification is correctly assigned to the respective document.

17. The computer program product of claim 15 , wherein the user interface comprises a dynamic user feedback guide for optimizing accuracy of the classification model through remedial actions, the remedial actions including adding more documents to the set of documents, adding more example documents to the set of example documents, or both.

18. The computer program product of claim 17 , wherein the dynamic user feedback guide is automatically activated and displayed without user intervention or user input.

19. The computer program product of claim 17 , wherein the dynamic user feedback guide is activated and displayed only when the accuracy of the classification model falls below a predetermined threshold.

20. The computer program product of claim 17 , wherein the dynamic user feedback gu

Assignees

• Crowdstrike Inc

Inventors

• Simard Charles-Olivier
• Bowyer Alex
• Leclerc Daniel
• Molloy Steve

Classifications

• G06F3/04842

  Selection of displayed objects or displayed text elements (G06F3/0482 takes precedence) · CPC title

• G06F3/0482

  Interaction with lists of selectable items, e.g. menus · CPC title

• G06N99/00

  Subject matter not provided for in other groups of this subclass · CPC title

• G06F16/285

  Clustering or classification · CPC title

• G06F16/258

  Data format conversion from or to a database · CPC title