Handling information source ingestion in a question answering system

Patent metadata

Field	Value
Publication number	US-9710522-B2
Application number	US-201615157440-A
Country	US
Kind code	B2
Filing date	May 18, 2016
Priority date	Aug 13, 2014
Publication date	Jul 18, 2017
Grant date	Jul 18, 2017

Abstract

A question-answering (QA) system first receives input questions. Each question is then assigned to a first question category of a plurality of question categories. The QA system then identifies a set of candidate answers to each question using a core information source. A set of confidence scores, including a confidence score for each candidate answer, is then calculated. The QA system then determines that the first set of confidence scores fails to satisfy confidence criteria. In response to this determination, an updated information source is ingested.

First claim

What is claimed is:

1. A system for handling a plurality of input questions, the system comprising:
a computer configured to:
prior to receiving the plurality of input questions, identify a first set of training candidate answers to each training question of a plurality of training questions using a core information source;
receive the plurality of input questions;
assign each question of the plurality of input questions to a first question category of a plurality of question categories;
identify a set of candidate answers to each input question of the plurality of input questions using the core information source;
calculate a first set of confidence scores, the first set of confidence scores including a confidence score for each candidate answer of each set of candidate answers;
provide the set of candidate answers to a user;
receive feedback from the user, the feedback including an indication that none of a set of candidate answers to a first input question of the plurality of input questions is accurate, and the feedback further including a correct answer to the first input question;
in response to determining that the first set of confidence scores fails to satisfy confidence criteria, ingest an updated information source that is separate from the feedback;
in response to the receiving the feedback, identify a new answer to the first input question using the updated information source;
compare the new answer identified using the updated information source to the correct answer included in the feedback;
determine, based on the comparing the new answer to the correct answer, that the new answer and the correct answer are not the same; and
in response to determining that the new answer and the correct answer are not the same, elect the updated information source without adding the updated information source to the core information source.

2. The system of claim 1 , wherein the computer is further configured to:
prior to the ingesting the updated information source, determine that the updated information source has an association with the first question category; and
select the updated information source for ingestion based on at least the association between the updated information source and the first question category.

3. The system of claim 1 , wherein the computer is further configured to:
receive a second plurality of input questions;
assign the second plurality of input questions to a second question category of the plurality of question categories;
identify a set of candidate answers to each input question of the second plurality of input questions using the core information source;
calculate a second set of confidence scores, the second set of confidence scores including a confidence score for each candidate answer of each set of candidate answers to each input question of the second plurality of input questions; and
determine that the second set of confidence scores satisfies the confidence criteria.

4. The system of claim 1 , wherein each question of the plurality of input questions are assigned to the first question category based on at least determining a type and a class for each question and analyzing the type and the class for each question.

5. A computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a computer to cause the computer to perform a method comprising:
prior to receiving a plurality of input questions, identifying a first set of training candidate answers to each training question of a plurality of training questions using a core information source;
receiving the plurality of input questions;
assigning each question of the plurality of input questions to a first question category of a plurality of question categories;
identifying a set of candidate answers to each input question of the plurality of input questions using the core information source;
calculating a first set of confidence scores, the first set of confidence scores including a confidence score for each candidate answer of each set of candidate answers;
providing the set of candidate answers to a user;
receiving feedback from the user, the feedback including an indication that none of a set of candidate answers to a first input question of the plurality of input questions is accurate, and the feedback further including a correct answer to the first input question;
in response to determining that the first set of confidence scores fails to satisfy confidence criteria, ingesting an updated information source that is separate from the feedback;
in response to the receiving the feedback, identifying a new answer to the first input question using the updated information source;
comparing the new answer identified using the updated information source to the correct answer included in the feedback;
determining, based on the comparing the new answer to the correct answer, that the new answer and the correct answer are not the same; and
in response to determining that the new answer and the correct answer are not the same, electing the updated information source without adding the updated information source to the core information source.

Assignees

• IBM

Inventors

• Allen Corville O
• Jamrog Daniel M
• Witherspoon Kristin A

Classifications

• G06F17/30554

  Physics · mapped topic

• G06F17/30528

  Physics · mapped topic

• G06F17/3043

  Physics · mapped topic

• G06F17/30867

  Physics · mapped topic

• G06F17/30873

  Physics · mapped topic