System and method for ground truth evaluation

Patent metadata

Field	Value
Publication number	US-10755182-B2
Application number	US-201615234676-A
Country	US
Kind code	B2
Filing date	Aug 11, 2016
Priority date	Aug 11, 2016
Publication date	Aug 25, 2020
Grant date	Aug 25, 2020

Abstract

A method for training a question answering system includes providing training questions to a question answering system executing on a computer and to a plurality of subject matter experts. The question answering system generates first answers to each training question. Second answers to the training questions are received from the subject matter experts. Feature scores for each of the first answers and the second answers are generated and compared across the second answers and the first answers. Each of the feature scores is representative of a quality of an answer that is indicative of relevance to a corresponding training question. Based on the comparing, a measure of consistency of the feature scores of the second answers is determined, and a measure of consistency of the feature scores of the second answers to the first answers is determined. The measures of consistency are transmitted to the subject matter experts.

First claim

What is claimed is:

1. A method comprising:
providing training questions to a question answering system executing on a computer;
producing, by the question answering system, candidate answers to each of the training questions;
providing the training questions and the candidate answers to a plurality of subject matter experts;
receiving, from each of the subject matter experts, relevance scores for each of the candidate answers, wherein the relevance score of a respective candidate answer indicates a relevance of that candidate answer to a respective training question that led to the candidate answer;
for each of the subject matter experts, generating a vector for relevance scores of the subject matter expert;
for the question answering system, generating a vector for relevance scores of the question answering system;
determining a metric based on the vectors of the subject matter experts and the vectors of the question answering system to determine whether agreement of relevance scores among the subject matter experts is higher than agreement between the subject matter experts and the question answering system; and
transmitting results of the determining to at least one of the subject matter experts.

2. The method of claim 1 , wherein the generating of the relevance scores comprises evaluating the candidate answers with respect to a plurality of scoring features comprising one or more scoring features selected from a group consisting of question type, focus, lexical answer type, sentence structure, term matching, and grammatical modifiers.

3. The method of claim 1 , further comprising configuring the question answering system to rank potential answers to a question based on the relevance scores of the subject matter experts.

4. The method of claim 1 , further comprising identifying, based on the results of the determining, a vector of at least one of the subject matter experts that exhibits less than a predetermined degree of similarity to the vectors of others of the subject matter experts.

5. The method of claim 1 , wherein determining the metric comprises:
determining a ratio of an average distance between each vector of the subject matter experts to an average distance between vectors of the subject matter experts and the question answering system.

6. A system comprising:
a question answering system executed by a computer;
a processor configured to communicate with the question answering system; and
a memory coupled to the processor, the memory encoded with instructions that when executed cause the processor to provide a training system for training the question answering system, the training system configured to:
provide training questions to the question answering system;
retrieve, from the question answering system, candidate answers to each of the training questions;
provide the training questions and the candidate answers to a plurality of subject matter experts;

receive, from each of the subject matter experts, relevance scores for each of the candidate answers, wherein the relevance score of a respective candidate answer indicates a relevance of that candidate answer to a respective training question that led to the candidate answer;
for each of the subject matter experts, generating a vector for relevance scores of the subject matter expert;
for the question answering system, generating a vector for relevance scores of the question answering system;
determining a metric based on the vectors of the subject matter experts and the vectors of the question answering system to determine whether agreement of relevance scores among the subject matter experts is higher than agreement between the subject matter experts and the question answering system; and
transmit results of the determining to one or more of the subject matter experts.

7. The system of claim 6 , wherein the training system is configured to evaluate the candidate answers with respect to a plurality of scoring features comprising one or more scoring features selected from a group consisting of question type, focus, lexical answer type, sentence structure, term matching, and grammatical modifiers.

8. The system of claim 6 , wherein the training system is configured to configure the question answering system to rank potential answers to a question based on the relevance scores of the subject matter experts.

9. The system of claim 6 , wherein the training system is configured to identify, based on the determining, a vector of at least one of the subject matter experts that exhibits less than a predetermined degree of similarity to the vectors of others of the subject matter experts.

10. The system of claim 6 , wherein the training system is configured to:
determine a ratio of an average distance between each vector of the subject matter experts to an average distance between vectors of the subject matter experts and the question answering system.

11. A computer program product for training a question answering system, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a computer to cause the computer to:
provide training questions to the question answering system;
retrieve, from the question answering system, candidate answers to each of the training questions;
provide the training questions and the candidate answers to a plurality of subject matter experts;
receive, from each of the subject matter experts, relevance scores for each of the candidate answers, wherein the relevance score of a respective candidate answer indicates a relevance of that candidate answer to a respective training question that led to the candidate answer;
for each of the subject matter experts, generating a vector for relevance scores of the subject matter expert;
for the question answering system, generating a vector for relevance scores of the question answering system;
determining a metric based on the vectors of the subject matter experts and the vectors of the question answering system to determine whether agreement of relevance scores among the subject matter experts is higher than agreement between the subject matter experts and the question answering system; and
transmit results of the determining to one or more of the subject matter experts.

12. The computer program product of claim 11 , wherein the program instructions are executable by the computer to cause the computer to evaluate the candidate answers with respect to a plurality of scoring features comprising one or more scoring features selected from a group consisting of question type, focus, lexical answer type, sentence structure, term matching, and grammatical modifiers.

13. The computer program product of claim 11 , wherein the program instructions are executable by the computer to cause the computer to configure the question answering system to rank potential answers to a question based on the relevance scores of the subject matter experts.

14. The computer program product of claim 11 , wherein the program instructions are executable by the computer to cause the computer to identify, based on the determining, a vector of at least one of the subject matter experts that exhibits less than a predetermined degree of similarity to the vectors of others of the subject matter experts.

15. The method of claim 1 , further comprising generating confidence scores indicating a confidence in accuracy of each of the candidate answers based on the relevance scores of the question answering system.

16. The method of claim 15 , further comprising ranking the c

Assignees

• IBM

Inventors

• Allen Corville O
• Freed Andrew R
• Kozhaya Joseph N
• Mansjur Dwi Sianto

Classifications

• G06N5/04 Primary

  Inference or reasoning models · CPC title