Interactive performance visualization of multi-class classifier

Patent metadata

Field	Value
Publication number	US-10460256-B2
Application number	US-201615232557-A
Country	US
Kind code	B2
Filing date	Aug 9, 2016
Priority date	Aug 9, 2016
Publication date	Oct 29, 2019
Grant date	Oct 29, 2019

Abstract

Methods, computer systems, computer-storage media, and graphical user interfaces are provided for improving performance of a multi-class classifier. An interactive graphical user interface includes an item representation display area that displays a plurality of item representations corresponding to a plurality of items processed by a multi-class classifier. The classifier's performance can be visualized using bidirectional bar graphs displaying true positives, false positives, and false negatives for each class.

First claim

What is claimed is:

1. A computing system comprising:
a processor; and
computer storage memory having computer-executable instructions stored thereon which, when executed by the processor, configure the computing system to improve a performance of a multi-class classifier, the computing system configured to:
receive performance metrics for the multi-class classifier that are generated by running test data through the multi-class classifier, the multi-class classifier trained to classify an item into one of a plurality of classes;
for a first class into which the multi-class classifier classifies items, determine an amount of false positives, an amount of true positives, and an amount of false negatives, wherein a false positive for the first class is an item in the test data that is labeled as a different class that is classified into the first class by the multi-class classifier, wherein a true positive is an item in the test data that is labeled as the first class and is classified into the first class by the multi-class classifier, and wherein a false negative is an item in the test data that is labeled as the first class but is classified into a different class by the multi-class classifier; and
for the first class, output for display on a bidirectional bar graph a representation of the amount of false positives and a representation of the amount of true positives on a first side of the bidirectional bar graph and a representation of the amount of false negatives on a second side of the bidirectional bar graph.

2. The computing system of claim 1 , wherein the representation of true positives is displayed in a color associated with the first class.

3. The computing system of claim 2 , wherein a portion of the representation of false positives is displayed in a color associated with a second class into which one or more items were classified.

4. The computing system of claim 1 , wherein the representation of the amount of true positives comprises multiple bars, wherein an individual bar from the multiple bars depicts an amount of true positive items that were assigned a confidence score within a confidence-score range associated with the individual bar.

5. The computing system of claim 1 , wherein the representation of the amount of false positives comprises multiple bars, wherein an individual bar from the multiple bars depicts an amount of false positive items that were assigned a confidence score within a confidence-score range associated with the individual bar.

6. The computing system of claim 1 , wherein the representation of the amount of false negatives comprises multiple bars, wherein an individual bar from the multiple bars depicts an amount of false negative items that were assigned a confidence score within a confidence-score range associated with the individual bar.

7. The computing system of claim 1 , further configured to output for display above the bidirectional bar graph a spark line showing confidence scores assigned to items labeled into the first class for each available class into which items can be assigned by the multi-class classifier.

8. A computer-implemented method for improving a multi-class classifier, the method comprising:
receiving performance metrics for the multi-class classifier that were generated by running test data through the multi-class classifier, the multi-class classifier trained to classify an item into one of a plurality of classes;
for each class in the plurality of classes, determining an amount of false positives for the class, an amount of true positives for the class, and an amount of false negatives for the class;
outputting for display a plurality of bidirectional bar graphs, each bidirectional bar graph associated with a different individual class in the plurality of classes, wherein a first bidirectional bar graph of the plurality of bidirectional bar graphs depicts performance for a first class, and wherein the first bidirectional bar graph depicts a representation of the amount of true positives for the first class in a first color associated with the first class on a first side of the first bidirectional bar graph, wherein the representation of the amount of true positives comprises a plurality of squares where each square represents an item correctly classified into the first class; and
receiving an interaction with an individual square and outputting for display a line graph that shows a confidence score calculated for each of the classes by the multi-class classifier for the item the individual square represents, and wherein the line graph intersects at least one of the plurality of bidirectional bar graphs.

9. The method of claim 8 , wherein the first bidirectional bar graph comprises a representation of an amount of false positives for a second class displayed on the first side of the first bidirectional bar graph in a second color associated with the second class.

10. The method of claim 9 , wherein the representation of false positives for the second class is displayed with a stripped pattern.

11. The method of claim 8 , wherein the representation of true positives comprises a plurality of stacked bars where each bar represents a group of items correctly classified into the first class.

12. The method of claim 8 , wherein the method further comprises concurrently displaying a portion of the performance metrics for items related to a portion of the bidirectional bar graph selected by the user.

13. The method of claim, further configured to output for display above the bidirectional bar graph a spark line showing confidence scores assigned to items labeled into the first class for each available class into which items can be assigned by the multi-class classifier.

14. The method of claim 8 , wherein the first bidirectional bar graph depicts a representation of a portion of the amount of false negatives for the first class in a third color associated with a third class on a second side of the first bidirectional bar graph, wherein the portion of false negatives represent an amount of items that are labeled in the first class and classified into the third class by the multi-class classifier.

15. The method of claim 14 , wherein the representation of the portion of the amount of false negatives are displayed as a geometric shape with a perimeter drawn in the third color.

16. One or more computer storage media having computer-executable instructions embodied thereon that, when executed by one or more processors, cause the one or more processors to perform a method for improving a multi-class classifier, the method comprising:
generating performance metrics for the multi-class classifier by running test data through the multi-class classifier, the multi-class classifier trained to classify an item into n classes, wherein n is an integer greater than 3;
for each of the n classes, determining an amount of false positives, an amount of true positives, and an amount of false negatives; and
for each of the n classes, outputting for display a bidirectional bar graph showing a representation of the amount of false positives and representation of the amount of true positives on a first side of the bidirectional bar graph and a first representation of the amount of false negatives on a second side of the bidirectional bar graph.

17. The media of claim 16 , wherein the representation of the amount of false positives comprises a first representation of items assigned a confidence score within a first range and a second representation of items assigned a confidence score within a second range.

18. The media of cl

Assignees

• Microsoft Technology Licensing Llc
• Microsot Tech Licensing Llc

Inventors

• Amershi Saleema A
• Lee Bongshin
• Suh Jina
• Williams Jason Douglas
• Ren Donghao

Classifications

• G06V10/776

  Validation; Performance evaluation · CPC title

• G06V10/764

  using classification, e.g. of video objects · CPC title

• G06T11/26

  Drawing of charts or graphs · CPC title

• G06T11/10

  Texturing; Colouring; Generation of textures or colours (retouching, inpainting or scratch removal G06T5/77) · CPC title

• G06F18/2431

  Multiple classes · CPC title