Cognitive visual debugger that conducts error analysis for a question answering system

Patent metadata

Field	Value
Publication number	US-10586161-B2
Application number	US-201514930872-A
Country	US
Kind code	B2
Filing date	Nov 3, 2015
Priority date	Nov 3, 2015
Publication date	Mar 10, 2020
Grant date	Mar 10, 2020

Abstract

A mechanism is provided in a data processing system for conducting error analysis for a question answering system. Responsive to the question answering system generating one or more candidate answers for an input question, wherein the one or more candidate answers are determined to be incorrect, the mechanism instantiates a plurality of instances of the question answering system with a modification to each instance. The mechanism provides the input question to each of the plurality of instances of the question answering system. The mechanism analyzes results from the plurality of instances of the question answering system to identify at least one modification that led to improved results. The mechanism presents a graphical output based on the analysis.

First claim

What is claimed is:

1. A method, in a data processing system, for conducting error analysis for a question answering system, the method comprising:
responsive to the question answering system generating one or more candidate answers for an input question, wherein the one or more candidate answers are determined to be incorrect, instantiating a plurality of instances of the question answering system with a modification to a pipeline of reasoning algorithms of each instance of the question answering system;
providing the input question to each of the plurality of instances of the question answering system executing in parallel;
analyzing results from the plurality of instances of the question answering system to identify at least one modification that led to improved results; and
presenting a graphical output based on the analysis.

2. The method of claim 1 , wherein analyzing results from the plurality of instances of the question answering system comprises:
comparing confidence scores of the one or more candidate answers to confidence scores received from the plurality of instances of the question answering system; and
identifying a modification to a given instance of the question answering system resulting in a higher confidence score.

3. The method of claim 1 , wherein analyzing results from the plurality of instances of the question answering system comprises determining steps to integrate changes to the question answering system.

4. The method of claim 1 , wherein the pipeline of reasoning algorithms of each instance of the question answering system comprises a plurality of stages.

5. The method of claim 4 , wherein the plurality of stages comprise a question and topic analysis stage, a question decomposition stage, a hypothesis generation stage, a hypothesis and evidence scoring stage, and a final confidence merging and ranking stage.

6. The method of claim 4 , wherein instantiating the plurality of instances of the question answering system further comprises performing the modification to the same stage of the pipeline of each instance of the question answering system.

7. A computer program product comprising a computer readable storage medium having a computer readable program stored therein, wherein the computer readable program, when executed on a computing device, causes the computing device to:
responsive to the question answering system generating one or more candidate answers for an input question, wherein the one or more candidate answers are determined to be incorrect, instantiate a plurality of instances of the question answering system with a modification to a pipeline of reasoning algorithms of each instance of the question answering system;
provide the input question to each of the plurality of instances of the question answering system executing in parallel;
analyze results from the plurality of instances of the question answering system to identify at least one modification that led to improved results; and
present a graphical output based on the analysis.

8. The computer program product of claim 7 , wherein analyzing results from the plurality of instances of the question answering system comprises:
comparing confidence scores of the one or more candidate answers to confidence scores received from the plurality of instances of the question answering system; and
identifying a modification to a given instance of the question answering system resulting in a higher confidence score.

9. The computer program product of claim 7 , wherein analyzing results from the plurality of instances of the question answering system comprises determining steps to integrate changes to the question answering system.

10. The computer program product of claim 7 , wherein the pipeline of reasoning algorithms of each instance of the question answering system comprises a plurality of stages.

11. The computer program product of claim 10 , wherein the plurality of stages comprise a question and topic analysis stage, a question decomposition stage, a hypothesis generation stage, a hypothesis and evidence scoring stage, and a final confidence merging and ranking stage.

12. The computer program product of claim 10 , wherein instantiating the plurality of instances of the question answering system further comprises performing the modification to the same stage of the pipeline of each instance of the question answering system.

13. An apparatus comprising:
a processor; and
a memory coupled to the processor, wherein the memory comprises instructions which, when executed by the processor, cause the processor to:
responsive to the question answering system generating one or more candidate answers for an input question, wherein the one or more candidate answers are determined to be incorrect, instantiate a plurality of instances of the question answering system with a modification to a pipeline of reasoning algorithms of each instance of the question answering system;
provide the input question to each of the plurality of instances of the question answering system executing in parallel;
analyze results from the plurality of instances of the question answering system to identify at least one modification that led to improved results; and
present a graphical output based on the analysis.

14. The apparatus of claim 13 , wherein analyzing results from the plurality of instances of the question answering system comprises:
comparing confidence scores of the one or more candidate answers to confidence scores received from the plurality of instances of the question answering system; and
identifying a modification to a given instance of the question answering system resulting in a higher confidence score.

15. The apparatus of claim 13 , wherein analyzing results from the plurality of instances of the question answering system comprises determining steps to integrate changes to the question answering system.

16. The apparatus of claim 13 , wherein the pipeline of reasoning algorithms of each instance of the question answering system comprises a plurality of stages.

17. The apparatus of claim 16 , wherein the plurality of stages comprise a question and topic analysis stage, a question decomposition stage, a hypothesis generation stage, a hypothesis and evidence scoring stage, and a final confidence merging and ranking stage.

18. The apparatus of claim 16 , wherein instantiating the plurality of instances of the question answering system further comprises performing the modification to the same stage of the pipeline of each instance of the question answering system.

Assignees

• IBM

Inventors

• Anderson John J
• Banks Kathryn V
• Fox Blake J
• Sinha Ravi S

Classifications

• G06N5/045 Primary

  Explanation of inference; Explainable artificial intelligence [XAI]; Interpretable artificial intelligence · CPC title