Augmenting Answer Keys with Key Characteristics for Training Question and Answer Systems

Patent metadata

Field	Value
Publication number	US-2016196504-A1
Application number	US-201514591413-A
Country	US
Kind code	A1
Filing date	Jan 7, 2015
Priority date	Jan 7, 2015
Publication date	Jul 7, 2016
Grant date	—

Abstract

Mechanisms are provided for implementing training logic for training a Question and Answer (QA) system. A training question, associated with an answer key, is received and processed by the QA system to generate a final answer to the training question and supporting evidence for the final answer based on a corpus of information. The supporting evidence is analyzed to identify one or more evidence attributes and a plurality of correct answer entries in the answer key are searched to identify a matching correct answer entry that matches the final answer. The matching correct answer entry in the answer key is augmented to include the one or more evidence attributes in an augmented answer key and the QA system is trained based on the augmented answer key.

First claim

What is claimed is:

1 . A method, in a data processing system comprising a processor and a memory and implementing training logic for training a Question and Answer (QA) system of the data processing system, the method comprising:
receiving, by the QA system, a training question; processing, by the QA system, the training question to generate a final answer to the training question and supporting evidence for the final answer based on a corpus of information; analyzing, by the training logic, the supporting evidence to identify one or more evidence attributes; searching, by the training logic, a plurality of correct answer entries in the answer key associated with the training question, to identify a matching correct answer entry that matches the final answer; augmenting the matching correct answer entry in the answer key to include the one or more evidence attributes in an augmented answer key; and training, by the training logic, the QA system based on the augmented answer key to generate a trained QA system.

2 . The method of claim 1 , wherein the one or more evidence attributes are one or more evidence attributes that differentiate conditions of supporting evidence under which a corresponding correct answer entry is more correct for the training question than other correct answer entries for the training question.

3 . The method of claim 1 , wherein the augmented answer key defines, for each training question, a spectrum of correct answers and corresponding circumstances under which each correct answer in the spectrum of correct answers is determined to be more correct than other correct answers in the spectrum of correct answers.

4 . The method of claim 1 , wherein the training question is a question for recommending a medical treatment for a patient, the one or more evidence attributes comprises characteristics of the patient, and wherein the final answer is a medical treatment recommendation.

5 . The method of claim 1 , wherein the one or more evidence attributes comprises one or more personal preferences of an individual represented in information in the corpus of information.

6 . The method of claim 1 , wherein the augmented answer key is one of a plurality of augmented answer keys, and wherein each augmented answer key is associated with a different set of evidence attributes from other augmented answer keys in the plurality of augmented answer keys.

7 . The method of claim 6 , further comprising:
receiving, by the QA system, one or more correlation characteristics associated with the training question, wherein training the QA system comprises selecting an augmented answer key from the plurality of augmented answer keys based on the one or more correlation characteristics submitted with the training question, and training the QA system using one or more correct answer entries for the training question stored in the selected augmented answer key.

8 . The method of claim 1 , wherein training the QA system comprises:
receiving, by the QA system, the training question; processing, by the QA system, the training question to generate a candidate final answer to the training question and supporting evidence for the candidate final answer based on a search of the corpus of information; selecting, by the QA system, a set of correct answers for the training question from the augmented answer key; comparing, by the QA system, the candidate final answer and supporting evidence for the candidate final answer to the set of correct answers and evidence attributes associated with each correct answer in the set of correct answers; and modifying an operation of the QA system based on results of the comparison.

9 . The method of claim 8 , wherein modifying the operation of the QA system comprises modifying weights associated with annotators of the QA system in a statistical model associated with the QA system.

10 . The method of claim 1 , further comprising:
answering, by the trained QA system, a subsequent question submitted to the trained QA system.

11 . A computer program product comprising a computer readable storage medium having a computer readable program stored therein, wherein the computer readable program, when executed on a data processing system implementing a Question and Answer (QA) system and associated training logic, causes the data processing system to:
receive, by the QA system, a training question; process, by the QA system, the training question to generate a final answer to the training question and supporting evidence for the final answer based on a corpus of information; analyze, by the training logic, the supporting evidence to identify one or more evidence attributes; search, by the training logic, a plurality of correct answer entries in the answer key associated with the training question, to identify a matching correct answer entry that matches the final answer; augment the matching correct answer entry in the answer key to include the one or more evidence attributes in an augmented answer key; and train, by the training logic, the QA system based on the augmented answer key to generate a trained QA system.

12 . The computer program product of claim 11 , wherein the one or more evidence attributes are one or more evidence attributes that differentiate conditions of supporting evidence under which a corresponding correct answer entry is more correct for the training question than other correct answer entries for the training question.

13 . The computer program product of claim 11 , wherein the augmented answer key defines, for each training question, a spectrum of correct answers and corresponding circumstances under which each correct answer in the spectrum of correct answers is determined to be more correct than other correct answers in the spectrum of correct answers.

14 . The computer program product of claim 11 , wherein the training question is a question for recommending a medical treatment for a patient, the one or more evidence attributes comprises characteristics of the patient, and wherein the final answer is a medical treatment recommendation.

15 . The computer program product of claim 11 , wherein the one or more evidence attributes comprises one or more personal preferences of an individual represented in information in the corpus of information.

16 . The computer program product of claim 11 , wherein the augmented answer key is one of a plurality of augmented answer keys, and wherein each augmented answer key is associated with a different set of evidence attributes from other augmented answer keys in the plurality of augmented answer keys.

17 . The computer program product of claim 16 , wherein the computer readable program further causes the data processing system to receive, by the QA system, one or more correlation characteristics associated with the training question, wherein the computer readable program further causes the data processing system to train the QA system at least by selecting an augmented answer key from the plurality of augmented answer keys based on the one or more correlation characteristics submitted with the training question, and training the QA system using one or more correct answer entries for the training question stored in the selected augmented answer key.

18 . The computer program product of claim 11 , wherein the computer readable program further causes the data processing system to train the QA system at least by:
receiving, by the QA system, the training question; processing, by the QA system, the training question to generate a candida

Assignees

• IBM

Inventors

• Allen Corville O
• Grev Christine A
• Stevens Richard J

Classifications

• G06N7/005

  Physics · mapped topic

• G06N99/005 Primary

  Physics · mapped topic

• G06N20/00 Primary

  Machine learning · CPC title

• G06N5/04

  Inference or reasoning models · CPC title

• G16H50/20

  for computer-aided diagnosis, e.g. based on medical expert systems · CPC title