Extensible validation framework for question and answer systems

Patent metadata

Field	Value
Publication number	US-11042794-B2
Application number	US-201715465682-A
Country	US
Kind code	B2
Filing date	Mar 22, 2017
Priority date	May 21, 2014
Publication date	Jun 22, 2021
Grant date	Jun 22, 2021

Abstract

Mechanisms are provided for validating a candidate answer to an input question. A candidate answer to an input question is identified using a natural language processing of the input question and a corpus of information from which the candidate answer is identified. A validator is selected to apply to the candidate answer based on a characteristic of a correct answer for the input question. The validator is applied to the candidate answer to evaluate whether or not criteria of the validator are met by the candidate answer. Validation information is generated based the evaluation of whether or not criteria of the validator are met by the candidate answer. The validation information is stored in a validation status object associated with the input question.

First claim

What is claimed is:

1. A method, in a data processing system comprising a processor and a memory, for validating a candidate answer to an input question, the method comprising:
registering, by the data processing system, a plurality of validators with a registry of a pluggable validator framework of the data processing system, wherein registration of each validator comprises mapping the validator to a corresponding correct answer characteristic;
generating, by the data processing system, a candidate answer to an input question using a natural language processing of the input question and a corpus of information from which the candidate answer is identified; and
after generation of the candidate answer:
selecting, by the data processing system, a validator, from the plurality of validators registered in the registry, to apply to the candidate answer based on a correspondence between a characteristic of a correct answer for the input question and a correct answer characteristic corresponding to the validator;
applying, by the data processing system, the validator to the candidate answer to evaluate whether or not one or more criteria of the validator are satisfied by the candidate answer;
generating, by the data processing system, validation information based on the evaluation of whether or not criteria of the validator are met by the candidate answer; and
storing, by the data processing system, the validation information in a validation status object associated with the input question.

2. The method of claim 1 , wherein selecting a validator to apply to the candidate answer based on a characteristic of the correct answer for the input question comprises:
performing a lookup operation in an answer key data structure for an entry corresponding to a question type of the input question, wherein the entry comprises at least one of an answer type or an answer extension type, and wherein the answer type is either a literal answer type or a regular expression answer type, and the answer extension type is an extended answer type other than a literal answer type or regular expression answer type; and
selecting the validator based on a correspondence between a correct answer characteristic corresponding to the validator and at least one of the answer type or the answer extension type.

3. The method of claim 1 , wherein the candidate answer comprises a wrapper having context information for the candidate answer, and wherein the validator operates on the context information in the wrapper of the candidate answer to determine if the one or more criteria of the validator are satisfied.

4. The method of claim 3 , wherein the wrapper comprises a confidence measure metric for the candidate answer, a rank of the candidate answer, and one or more identifiers of supporting evidence passages that provide support for the candidate answer being a correct answer for the input question.

5. The method of claim 1 , wherein the validator is one of an applied plurality of validators selected to be applied to the candidate answer based on the answer key data structure, and wherein the validation status object stores results of applying each of the plurality of validators to the candidate answer.

6. The method of claim 5 , wherein the validation status object further stores, for at least one validator of the applied plurality of validators, a log of operations performed by the at least one validator and corresponding results of the operations performed by the at least one validator.

7. The method of claim 1 , wherein the validator evaluates the one or more criteria against at least one supporting evidence passage associated with the candidate answer or a performance characteristic associated with at least one stage of a question and answer pipeline implemented by the data processing system.

8. The method of claim 1 , wherein selecting the validator comprises:
selecting the validator from the plurality of validators registered with the pluggable validator framework based on a first lookup of one of an answer type or answer extension type corresponding to the input question in the answer key data structure;
performing a second lookup of the answer type or answer extension type in the pluggable validator framework to identify a correlation of the validator with the answer type or answer extension type; and
selecting the validator based on results of performing the second lookup.

9. The method of claim 1 , wherein the method is implemented during a training operation for training the data processing system to answer input questions, and wherein the validation status object is processed by the data processing system to train the data processing system to answer questions similar to the input question.

10. A computer program product comprising a computer readable storage medium having a computer readable program stored therein, wherein the computer readable program, when executed on a data processing system, causes the data processing system to:
register a plurality of validators with a registry of a pluggable validator framework of the data processing system, wherein registration of each validator comprises mapping the validator to a corresponding correct answer characteristic;
generate a candidate answer to an input question using a natural language processing of the input question and a corpus of information from which the candidate answer is identified; and
after generation of the candidate answer:
select a validator, from the plurality of validators registered in the registry, to apply to the candidate answer based on a correspondence between a characteristic of a correct answer for the input question and a correct answer characteristic corresponding to the validator;
apply the validator to the candidate answer to evaluate whether or not one or more criteria of the validator are satisfied by the candidate answer;
generate validation information based on the evaluation of whether or not criteria of the validator are met by the candidate answer; and
store the validation information in a validation status object associated with the input question.

11. The computer program product of claim 10 , wherein the computer readable program further causes the data processing system to select a validator to apply to the candidate answer based on a characteristic of the correct answer for the input question at least by:
performing a lookup operation in an answer key data structure for an entry corresponding to a question type of the input question, wherein the entry comprises at least one of an answer type or an answer extension type, and wherein the answer type is either a literal answer type or a regular expression answer type and the answer extension type is an extended answer type other than a literal answer type or regular expression answer type; and
selecting the validator based on a correspondence between a correct answer characteristic corresponding to the validator and at least one of the answer type or the answer extension type.

12. The computer program product of claim 10 , wherein the candidate answer comprises a wrapper having context information for the candidate answer, and wherein the validator operates on the context information in the wrapper of the candidate answer to determine if the one or more criteria of the validator are satisfied.

13. The computer program product of claim 12 , wherein the wrapper comprises a confidence measure metric for the candidate answer, a rank of the candidate answer, and one or more identifiers of supporting evidence passages that provide support for the candidate answer being a correct answer for the input question.

14.

Assignees

• IBM

Inventors

• Allen Corville O
• Bennett Ian M
• Bittner Torsten
• Mueller Kay

Classifications

• G06N7/01

  Probabilistic graphical models, e.g. probabilistic networks · CPC title

• G06N5/041 Primary

  Abduction · CPC title

• G06F16/3329

  Natural language query formulation · CPC title

• G06F16/2365

  Ensuring data consistency and integrity · CPC title

• G06N3/006 Primary

  based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO] · CPC title