System support for evaluation consistency

Patent metadata

Field	Value
Publication number	US-9633003-B2
Application number	US-201213718495-A
Country	US
Kind code	B2
Filing date	Dec 18, 2012
Priority date	Dec 18, 2012
Publication date	Apr 25, 2017
Grant date	Apr 25, 2017

Abstract

A system and computer product for validating the consistency between quantitative and natural language textual evaluations. An example method involves computing a numeric score for a textual evaluation, comparing the numeric score to a quantitative evaluation, and producing a rating based on the similarity of the two evaluations.

First claim

What is claimed is:

1. A system for validating the consistency between quantitative and natural language textual evaluations of an item, the system comprising:
a computer processor;
receiving a quantitative evaluation of the item from an evaluator;
receiving a natural language textual evaluation of the item from the evaluator;
a computing module coupled to the computer processor configured to compute a numeric score for the natural language textual evaluation based on a textual analysis of the natural language textual evaluation;
a comparing module coupled to the computer processor configured to determine the degree to which the numeric score and the quantitative evaluation are consistent with each other;
a calculating module coupled to the computer processor configured to calculate a similarity rating from the comparison of the quantitative evaluation and the numeric score; and
an alert module to display an alert in a user interface to the evaluator when an inconsistency between the numeric score and the quantitative evaluation is above a defined level.

2. The system of claim 1 , wherein the computing module is further configured to compute the numeric score by performing a sentiment analysis on the natural language textual evaluation, the sentiment analysis includes determining the polarity of written text by locating terms that are rated with positive polarity and other terms having negative polarities, and summing the results to get the numeric score.

3. The system of claim 1 , wherein the computing module is further configured to compute multiple numeric scores for the natural language textual evaluation based on multiple predetermined evaluation criteria.

4. The system of claim 1 , further comprising an input module configured to input text in separate windows based on different evaluation criteria.

5. The system of claim 1 , further comprising an input module configured to input multiple quantitative evaluations based on different evaluation criteria.

6. The system of claim 1 , wherein the comparing module is further configured to produce the similarity rating using any one of Pearson's correlation and cosine similarity.

7. The system of claim 1 , further comprising a display module configured to display the similarity rating to the evaluator upon submission of the evaluation and provide the evaluator an opportunity to adjust his evaluations.

8. The system of claim 1 , further comprising an input module configured to increase the accuracy of the similarity comparison by allowing the evaluator to report that a displayed similarity rating is inaccurate, and using the report for training a component that produces the similarity rating.

9. The system of claim 1 , further comprising a display module configured to display at least one numeric score as at least one of an exact rating and probability distribution.

10. A computer program product for validating the consistency between quantitative and natural language textual evaluations of an item, the computer program product comprising:
a non-transitory computer readable storage medium having computer readable program code embodied therewith, the computer readable program code configured to:
receive a quantitative evaluation of the item from an evaluator;
receive a natural language textual evaluation of the item from the evaluator;
compute a numeric score for the natural language textual evaluation based on a textual analysis of the for the natural language textual evaluation;
compare the numeric score to the quantitative evaluation;
calculate a similarity rating from the comparison of the quantitative evaluation and the numeric score; and
alert the evaluator when an inconsistency between the numeric score and the quantitative evaluation is above a defined level by displaying an alert in a user interface.

11. The computer program product of claim 10 , wherein the computer readable program code to compute the numeric score includes the computer readable program code to perform a sentiment analysis on the natural language textual evaluation, the sentiment analysis includes determining the polarity of written text by locating terms that are rated with positive polarity and other terms having negative polarities, and summing the results to get the numeric score.

12. The computer program product of claim 10 , further comprising the computer readable program code to compute multiple numeric scores created for the natural language textual evaluation based on multiple predetermined evaluation criteria.

13. The computer program product of claim 10 , further comprising the computer readable program code to receive text inputted in separate windows based on different evaluation criteria.

14. The computer program product of claim 10 , further comprising the computer readable program code to receive multiple quantitative evaluations based on different evaluation criteria.

Assignees

• IBM

Inventors

• Soroker Danny
• Weisz Justin D

Classifications

• G06F40/00

  Handling natural language data (speech analysis or synthesis, speech recognition G10L) · CPC title

• G06F40/30 Primary

  Semantic analysis · CPC title

• G06F40/20

  Natural language analysis (semantic analysis of natural language G06F40/30) · CPC title

• G06F17/20

  Physics · mapped topic

• G06F17/27

  Physics · mapped topic