User collaboration for answer generation in question and answer system

Patent metadata

Field	Value
Publication number	US-9621601-B2
Application number	US-201313853261-A
Country	US
Kind code	B2
Filing date	Mar 29, 2013
Priority date	Mar 29, 2013
Publication date	Apr 11, 2017
Grant date	Apr 11, 2017

Abstract

Mechanisms are provided, in association with a question and answer system, for generating answers to an input question. An input question is received and processed to generate at least one query. The at least one query is applied to the corpus to generate a set of candidate answers and corresponding selections of evidence portions of the corpus of information providing evidence in support of the at least one candidate answer being a correct answer for the input question. A graphical user interface (GUI) is output comprising a first GUI sub-section comprising the set of candidate answers, and a second GUI sub-section comprising the evidence portions of the corpus. User input modifying at least one of set of candidate answers or the evidence portions of the corpus of information is received via the GUI and an operation of the data processing system is adjusted based on the user input.

First claim

What is claimed is:

1. A method, in a data processing system comprising a processor and a memory configured to implement a question and answer system (QA), for generating answers to an input question, comprising:
receiving, in the data processing system, the input question;
processing, by the data processing system, the input question to generate at least one query for application to a corpus of information;
applying, by the data processing system, the at least one query to the corpus of information to generate a set of candidate answers to the input question and corresponding selections of evidence portions of the corpus of information providing evidence in support of the at least one candidate answer in the set of candidate answers being a correct answer for the input question;
outputting, via a graphical user interface (GUI), the set of candidate answers in a first GUI sub-section, and the evidence portions in a second GUI sub-section, for evaluation by a user;
receiving, via the graphical user interface, user input modifying at least one of the set of candidate answers or the evidence portions; and
adjusting an operation of the data processing system based on the user input, wherein the user input specifies an action for adding a new candidate answer to the at least one candidate answer, and wherein adjusting the operation of the data processing system based on the user input comprises:
adding the new candidate answer to the at least one candidate answer;
automatically submitting the new candidate answer to the QA system for evaluation in response to the addition of the new candidate answer; and
automatically updating the first GUI sub-section and second GUI sub-section based on results of the evaluation of the new candidate answer by the QA system.

2. The method of claim 1 , wherein, the set of candidate answers comprises at least one candidate answer, the user input selects a GUI element for removing a candidate answer from the set of candidate answers, and wherein adjusting the operation of the data processing system based on the user input comprises:
removing the candidate answer from the first GUI sub-section; and
automatically removing an evidence portion of the second GUI sub-section corresponding to the removed candidate answer.

3. The method of claim 2 , wherein the first GUI sub-section is organized based on confidence scores associated with the at least one candidate answer, and wherein adjusting the operation of the data processing system further comprises updating the first GUI sub-section to modify the organization of the first GUI sub-section in view of the removal of the candidate answer from the first GUI sub-section.

4. The method of claim 1 , wherein the user input specifying the action for adding the new candidate answer is a user input to a free-form field for user input of a textual representation of a new candidate answer in a free-form manner.

5. The method of claim 1 , wherein the user input specifying the action for adding the new candidate answer is a user input selecting a sub-portion of one evidence portion output in the second GUI sub-section, and wherein the selected sub-portion of the one evidence portion is the new candidate answer.

6. The method of claim 5 , wherein the selected sub-portion of the evidence portion in the second GUI sub-section is one of a word, a phrase, or a sentence of the evidence portion.

7. The method of claim 1 , wherein the user input further comprises a selection of a drill-down GUI element associated with an evidence portion in the second GUI sub-section, and wherein adjusting an operation of the data processing system based on the user input comprises:
retrieving source document information for the evidence portion associated with the drill-down GUI element; and
outputting the source document information, via the GUI, for evaluation by the user.

8. The method of claim 7 , wherein the source document information comprises content of the source document in close proximity to the evidence portion within the source document.

9. The method of claim 1 , wherein the user input further specifies an action for removing a selected evidence portion from the second GUI sub-section, and wherein adjusting an operation of the data processing system comprises:
removing the selected evidence portion from the second GUI sub-section;
submitting the removal of the selected evidence portion to the QA system to re-calculate a confidence score for a candidate answer associated with the selected evidence portion;
re-calculating the confidence score for the candidate answer in view of the removal of the selected evidence portion to generate a new confidence score for the candidate answer; and
updating the first GUI sub-section based on the new confidence score for the candidate answer.

10. The method of claim 1 , wherein evidence portions in the second GUI sub-section comprise a graphical indicator of a relevance score indicating a relevance of the evidence portion to a corresponding candidate answer in the first GUI sub-section.

11. The method of claim 10 , wherein the relevance score for each evidence portion is calculated using a statistical machine learning model based on ground truth data indicating whether the corresponding evidence portion is justifying or not-justifying of the candidate answer.

12. A computer program product comprising a non-transitory computer readable medium having a computer readable program stored therein, wherein the computer readable program, when executed on a data processing system implementing a question and answer system (QA), causes the data processing system to:
receive an input question;
process the input question to generate at least one query for application to a corpus of information;
apply the at least one query to the corpus of information to generate a set of candidate answers to the input question and corresponding selections of evidence portions of the corpus of information providing evidence in support of the at least one candidate answer being a correct answer for the input question;
output, via a graphical user interface (GUI), the set of candidate answer in a first GUI sub-section, and the evidence portions in a second GUI sub-section, for evaluation by a user;
receive, via the graphical user interface, user input modifying at least one of the set of candidate answers or the evidence portions; and
adjust an operation of the data processing system based on the user input, wherein the user input specifies an action for adding a new candidate answer to the at least one candidate answer, and wherein adjusting the operation of the data processing system based on the user input comprises:
adding the new candidate answer to the at least one candidate answer;
automatically submitting the new candidate answer to the QA system for evaluation in response to the addition of the new candidate answer; and
automatically updating the first GUI sub-section and second GUI sub-section based on results of the evaluation of the new candidate answer by the QA system.

13. The computer program product of claim 12 , wherein, the set of candidate answers comprises at least one candidate answer, the user input selects a GUI element for removing a candidate answer from the set of candidate answers, and wherein adjusting the operation of the data processing system based on the user input comprises:
removing the candidate answer from the first GUI sub-section; and
automatically removing an evidence portion of the second GUI sub-section corresponding to the removed candidate answer.

14. The computer program product of claim 13 , wherein the first GU

Assignees

• IBM

Inventors

• Johnson Jr Robert C
• Lewis Scott M
• Whitley Michael D
• Yates Robert L

Classifications

• G06Q10/101 Primary

  Collaborative creation, e.g. joint development of products or services · CPC title

• H04L65/403 Primary

  Arrangements for multi-party communication, e.g. for conferences (data switching systems for conference H04L12/18; arrangements for connecting several subscribers to a common circuit, i.e. affording conference facilities H04M3/56; television conferencing systems H04N7/15) · CPC title