Expanding an answer key to verify a question and answer system

Patent metadata

Field	Value
Publication number	US-10567552-B2
Application number	US-201414332573-A
Country	US
Kind code	B2
Filing date	Jul 16, 2014
Priority date	Dec 17, 2013
Publication date	Feb 18, 2020
Grant date	Feb 18, 2020

Abstract

A method for expanding an answer key to verify a question and answer system is provided in the illustrative embodiments. A definition is constructed of an extended answer type. The extended answer type represents an answer type of an unrepresented answer. The unrepresented answer is unrepresented in the answer key as a valid response to a question in a set of valid responses to the question in the answer key. The extended answer type is created in the answer key according to the definition. The extended answer type is populated such that the unrepresented answer becomes as additional valid response to the question, the creating and the populating extending the answer key to form an extended answer key. The populated extended answer type in the extended answer key is used to verify that a generated answer from the Q and A system is correct.

First claim

What is claimed is:

1. A method for expanding an answer key to verify a question and answer (Q and A) system, the method comprising:
executing an answer verification application in conjunction with an artificial intelligence (AI) application, the AI application executing as a part of the Q and A system in a data processing system, the AI application configured to compute a natural language (NL) answer in response to an NL question input into the Q and A system by (i) parsing the NL question into a major feature, (ii) constructing a query from the major feature, (iii) applying the query to a corpus to select a portion of the corpus that is responsive to the NL question, (iv) executing a reasoning algorithm to perform a deep analysis of a language of the NL question and a language of the portion of the corpus, the deep analysis returning a score, the score being indicative of an extent to which a potential response is inferred by the NL question, (v) weighting the score against a statistical model to determine that the potential response is strongest in a set of potential responses, and (vi) outputting the potential response as a generated answer from the Q and A system, wherein the executing the answer verification application evaluates an accuracy of the Q and A system, the executing the answer verification application comprising:
constructing, a definition of an extended answer type, wherein the extended answer type represents an answer type of an unrepresented answer, wherein the unrepresented answer is unrepresented in the answer key as a valid response to the NL question in a set of valid responses to the NL question in the answer key;
creating the extended answer type in the answer key according to the definition;
populating the extended answer type such that the unrepresented answer becomes as additional valid response to the question, the creating and the populating extending the answer key to form an extended answer key; and
using the populated extended answer type in the extended answer key to verify that the generated answer from the Q and A system is correct.

2. The method of claim 1 , wherein the generated answer is incorrect according to an existing answer type in the answer key.

3. The method of claim 1 , further comprising:
forming the definition by modifying an abstract definition;
constructing a second definition of a second extended answer type, wherein the second extended answer type represents an answer type of a second unrepresented answer.

4. The method of claim 3 , wherein the abstract definition is used as a placeholder in the extended answer key for a third extended answer type.

5. The method of claim 1 , wherein the unrepresented answer becomes the additional valid answer because a computation using the populated extended answer type results in the additional valid response.

6. The method of claim 1 , wherein the unrepresented answer becomes the additional valid answer because a logic described in a metadata of the extended answer key computes to make the unrepresented answer the additional valid response.

7. The method of claim 1 , wherein the unrepresented answer becomes the additional valid answer by being a member of a range of values specified in the populated extended answer type.

8. The method of claim 1 , wherein the unrepresented answer becomes the additional valid answer by being a member of a range of values, wherein the range of values is computed using logic in a metadata of the extended answer key.

9. The method of claim 1 , further comprising:
configuring the extended answer type with a changeable condition, wherein a value of the additional valid answer changes when the condition changes.

10. The method of claim 1 , wherein the extended answer key in an Extensible Markup Language (XML) document, wherein the definition is an XML structure.

11. The method of claim 1 , wherein the question comprise a sentence in natural language, wherein the Q and A system is configured to respond to the natural language question.

12. The method of claim 1 , wherein the question comprise a sentence in natural language, wherein the Q and A system is configured to respond to the natural language question using another extended answer type to represent answers one of (i) based on a cultural reference, (ii) in a particular language, and (iii) from a set of synonyms.

Assignees

• IBM

Inventors

• Allen Corville Orain
• Bennett Ian Michael
• Bittner Torsten
• Muller Kay

Classifications

• G09B7/00

  Electrically-operated teaching apparatus or devices working with questions and answers (mechanically operated G09B3/00; computing arrangements G06F) · CPC title

• G06N5/04

  Inference or reasoning models · CPC title

• G09B7/02

  of the type wherein the student is expected to construct an answer to the question which is presented or wherein the machine gives an answer to the question presented by a student · CPC title

• G06F9/453

  Help systems · CPC title

• H04L67/125

  involving control of end-device applications over a network · CPC title