Extensible Validation Framework for Question and Answer Systems

Patent metadata

Field	Value
Publication number	US-2017193350-A1
Application number	US-201715465682-A
Country	US
Kind code	A1
Filing date	Mar 22, 2017
Priority date	May 21, 2014
Publication date	Jul 6, 2017
Grant date	—

Abstract

Mechanisms are provided for validating a candidate answer to an input question. A candidate answer to an input question is identified using a natural language processing of the input question and a corpus of information from which the candidate answer is identified. A validator is selected to apply to the candidate answer based on a characteristic of a correct answer for the input question. The validator is applied to the candidate answer to evaluate whether or not criteria of the validator are met by the candidate answer. Validation information is generated based the evaluation of whether or not criteria of the validator are met by the candidate answer. The validation information is stored in a validation status object associated with the input question.

First claim

1 . A method, in a data processing system comprising a processor and a memory, for validating a candidate answer to an input question, the method comprising:
generating, by the data processing system, a candidate answer to an input question using a natural language processing of the input question and a corpus of information from which the candidate answer is identified; selecting, by the data processing system, a validator to apply to the candidate answer based on a characteristic of a correct answer for the input question; applying, by the data processing system, the validator to the candidate answer to evaluate whether or not one or more criteria of the validator are satisfied by the candidate answer; generating, by the data processing system, validation information based the evaluation of whether or not criteria of the validator are met by the candidate answer; and storing, by the data processing system, the validation information in a validation status object associated with the input question.

2 . The method of claim 1 , wherein selecting a validator to apply to the candidate answer based on a characteristic of the correct answer for the input question comprises:
performing a lookup operation in an answer key data structure for an entry corresponding to a question type of the input question, wherein the entry comprises at least one of an answer type or an answer extension type, and wherein the answer type is either a literal answer type or a regular expression answer type and the answer extension type is an extended answer type other than a literal answer type or regular expression answer type; and selecting the validator based on a correspondence between the validator and at least one of the answer type or the answer extension type.

3 . (canceled)

4 . The method of claim 1 , wherein the candidate answer comprises a wrapper having context information for the candidate answer, and wherein the validator operates on the context information in the wrapper of the candidate answer to determine if the one or more criteria of the validator are satisfied.

5 . The method of claim 4 , wherein the wrapper comprises a confidence measure metric for the candidate answer, a rank of the candidate answer, and one or more identifiers of supporting evidence passages that provide support for the candidate answer being a correct answer for the input question.

6 . The method of claim 1 , wherein the validator is one of a plurality of validators selected to be applied to the candidate answer based on the answer key data structure, and wherein the validation status object stores results of applying each of the plurality of validators to the candidate answer.

7 . The method of claim 6 , wherein the validation status object further stores, for at least one validator of the plurality of validators, a log of operations performed by the at least one validator and corresponding results of the operations performed by the at least one validator.

8 . The method of 1 , wherein the validator evaluates the one or more criteria against at least one supporting evidence passage associated with the candidate answer or a performance characteristic associated with at least one stage of a question and answer pipeline implemented by the data processing system.

9 . The method of 1 , further comprising:
registering each validator, in a plurality of validators, with a pluggable validator framework, wherein registration of each validator comprises mapping the validator to a corresponding answer type or answer extension type, and wherein selecting the validator comprises: selecting the validator from the plurality of validators registered with the pluggable validator framework based on a first lookup of one of an answer type or answer extension type corresponding to the input question in the answer key data structure; performing a second lookup of the answer type or answer extension type in the pluggable validator framework to identify a correlation of the validator with the answer type or answer extension type; and selecting the validator based on results of performing the second lookup.

10 . The method of claim 1 , wherein the method is implemented during a training operation for training the data processing system to answer input questions, and wherein the validation status object is processed by the data processing system to train the data processing system to answer questions similar to the input question.

11 . A computer program product comprising a computer readable storage medium having a computer readable program stored therein, wherein the computer readable program, when executed on a data processing system, causes the data processing system to:
generate a candidate answer to an input question using a natural language processing of the input question and a corpus of information from which the candidate answer is identified; select a validator to apply to the candidate answer based on a characteristic of a correct answer for the input question; apply the validator to the candidate answer to evaluate whether or not one or more criteria of the validator are satisfied by the candidate answer; generate validation information based the evaluation of whether or not criteria of the validator are met by the candidate answer; and store the validation information in a validation status object associated with the input question.

12 . The computer program product of claim 11 , wherein the computer readable program further causes the data processing system to select a validator to apply to the candidate answer based on a characteristic of the correct answer for the input question at least by:
performing a lookup operation in an answer key data structure for an entry corresponding to a question type of the input question, wherein the entry comprises at least one of an answer type or an answer extension type, and wherein the answer type is either a literal answer type or a regular expression answer type and the answer extension type is an extended answer type other than a literal answer type or regular expression answer type; and selecting the validator based on a correspondence between the validator and at least one of the answer type or the answer extension type.

13 . (canceled)

14 . The computer program product of claim 11 , wherein the candidate answer comprises a wrapper having context information for the candidate answer, and wherein the validator operates on the context information in the wrapper of the candidate answer to determine if the one or more criteria of the validator are satisfied.

15 . The computer program product of claim 14 , wherein the wrapper comprises a confidence measure metric for the candidate answer, a rank of the candidate answer, and one or more identifiers of supporting evidence passages that provide support for the candidate answer being a correct answer for the input question.

16 . The computer program product of claim 11 , wherein the validator is one of a plurality of validators selected to be applied to the candidate answer based on the answer key data structure, and wherein the validation status object stores results of applying each of the plurality of validators to the candidate answer.

17 . The computer program product of claim 16 , wherein the validation status object further stores, for at least one validator of the plurality of validators, a log of operations performed by the at least one validator and corresponding results of the operations performed by the at least one validator.

18 . The computer program product of 11 , wherein t

Assignees

• IBM

Inventors

• Allen Corville O
• Bennett Ian M
• Bittner Torsten
• Mueller Kay

Classifications

• G06N7/01

  Probabilistic graphical models, e.g. probabilistic networks · CPC title

• G06N5/041 Primary

  Abduction · CPC title

• G06F16/3329

  Natural language query formulation · CPC title

• G06F40/30

  Semantic analysis · CPC title

• G06F16/2365

  Ensuring data consistency and integrity · CPC title