Automatic Generation of Training Cases and Answer Key from Historical Corpus

Patent metadata

Field	Value
Publication number	US-2019325347-A1
Application number	US-201916460368-A
Country	US
Kind code	A1
Filing date	Jul 2, 2019
Priority date	Nov 25, 2014
Publication date	Oct 24, 2019
Grant date	—

Abstract

Mechanisms are provided for training and operating a Question and Answer (QA) system pipeline. A corpus of information is received which comprises historical data to which one or more filter criteria are applied to extract filtered historical data relevant to a training objective for training the QA system pipeline. Attribute data, action data, and temporal characteristic data are captured from the filtered historical data. An answer key entry is automatically generated in an automatically generated training answer key data structure based on the attribute data, action data, and temporal characteristic data. The correct answer associated with the answer key entry is an action specified by the action data. The temporal characteristic data provides a historical context for the answer key entry. The QA system pipeline is trained using the automatically generated training answer key data structure.

First claim

1 . A method, in a data processing system having a processor and a memory, wherein the memory comprises instructions which are executed by the processor to cause the processor to implement a training engine for generating training cases and an answer key from a historical corpus for training a Question and Answer (QA) system, the method comprising:
receiving, by corpus ingestion logic executing within the training engine, a corpus of information comprising historical data; automatically applying, by filtering logic executing within the training engine, one or more filter criteria to the historical data to extract filtered historical data relevant to a training objective for training the QA system pipeline; automatically capturing, by system answer key and training case generation logic executing within the training engine, attribute data, action data, and temporal characteristic data from the filtered historical data; automatically generating, by the answer key and training case generation logic, an answer key entry in an automatically generated training answer key data structure based on the attribute data, action data, and temporal characteristic data; and training, by the training engine, a QA system pipeline for the QA system using the automatically generated training answer key data structure.

2 . The method of claim 1 , wherein a correct answer associated with the answer key entry is an action specified by the action data, and wherein the temporal characteristic data provides a historical context for the answer key entry.

3 . The method of claim 1 , wherein the one or more filter criteria comprises at least one of a source filter criterion used to select historical data associated with specific sources of information in the corpus of information, a temporal filter criterion that is used to select information in the corpus of information that is more contemporary, or a confidence filter criterion that is used to select historical data with at least a specified level of confidence associated with the historical data.

4 . The method of claim 1 , wherein the action data of the answer key entry specifies a correct answer to a question, the attribute data of the answer key entry comprises question features for correlating the question to the correct answer, and the temporal characteristic data specifies a historical date or time at which the answer was considered correct for the question.

5 . The method of claim 4 , wherein training the QA system pipeline using the automatically generated training answer key data structure comprises:
receiving, by the QA system pipeline, a training case comprising a training question for processing by the QA system pipeline and a training case temporal characteristic; filtering, by the QA system pipeline, the corpus of information based on the training case temporal characteristic to thereby generate a temporally filtered sub-corpus; and processing, by the QA system pipeline, the training question based on the temporally filtered sub-corpus to generate an answer to the training question.

6 . The method of claim 5 , wherein training the QA system pipeline further comprises:
comparing, by the QA system pipeline, the answer to the training question with a correct answer in a corresponding training answer key entry of the training answer key data structure; and modifying, by the QA system pipeline, an operation of the QA system pipeline based on results of the comparing.

7 . The method of claim 6 , wherein the corresponding training answer key entry of the training answer key data structure is a training answer key entry having attributes matching attributes of the training question and a temporal characteristic matching the training case temporal characteristic.

8 . The method of claim 5 , wherein training the QA system pipeline comprises generating, through a machine learning process, a trained model, having weights to be applied to annotation logic of the QA system pipeline, based on a degree of correspondence between the answer to the training question and a correct answer specified in a training answer key entry corresponding to the training question and training case temporal characteristic of the training case.

9 . The method of claim 1 , wherein the corpus of information comprises financial investment information, and wherein the filtered historical data comprises entries in the financial investment information directed to investment decisions made by financial investors on or prior to a date or time specified in the temporal characteristic.

10 . The method of claim 1 , wherein the corpus of information comprises patient medical records, and wherein the filtered historical data comprises entries in patient medical records directed to healthcare services provided to patients on or prior to a date or time specified in the temporal characteristic.

11 . A computer program product comprising a computer readable storage medium having a computer readable program stored therein, wherein the computer readable program, when executed on a computing device, causes the computing device to implement a training engine for generating training cases and an answer key from a historical corpus for training a Question and Answer (OA) system, wherein the computer readable program further causes the computing device to:
receive, by corpus ingestion logic executing within the training engine, a corpus of information comprising historical data; automatically apply, by filtering logic executing within the training engine, one or more filter criteria to the historical data to extract filtered historical data relevant to a training objective for training a Question and Answer (QA) system pipeline; automatically capture, by answer key and training case generation logic executing within the training engine, attribute data, action data, and temporal characteristic data from the filtered historical data; automatically generate, by the answer key and training case generation logic, an answer key entry in an automatically generated training answer key data structure based on the attribute data, action data, and temporal characteristic data; and train, by the training engine, a QA system pipeline for the QA system using the automatically generated training answer key data structure.

12 . The computer program product of claim 11 , wherein a correct answer associated with the answer key entry is an action specified by the action data, and wherein the temporal characteristic data provides a historical context for the answer key entry.

13 . The computer program product of claim 11 , wherein the one or more filter criteria comprises at least one of a source filter criterion used to select historical data associated with specific sources of information in the corpus of information, a temporal filter criterion that is used to select information in the corpus of information that is more contemporary, or a confidence filter criterion that is used to select historical data with at least a specified level of confidence associated with the historical data.

14 . The computer program product of claim 11 , wherein the action data of the answer key entry specifies a correct answer to a question, the attribute data of the answer key entry comprises question features for correlating the question to the correct answer, and the temporal characteristic data specifies a historical date or time at which the answer was considered correct for the question.

15 . The computer program product of claim 14 , wherein the computer readable program further causes the computing device to train the QA system pipeline using the aut

Assignees

• IBM

Inventors

• Allen Corville O
• Grev Christine A
• Stevens Richard J
• Whaley Kathryn L

Classifications

• G16H10/60 Primary

  for patient-specific data, e.g. for electronic patient records · CPC title

• G06N20/00 Primary

  Machine learning · CPC title

• G06N5/02 Primary

  Knowledge representation; Symbolic representation · CPC title