Supporting evidence retrieval for complex answers

Patent metadata

Field	Value
Publication number	US-10810215-B2
Application number	US-201715844289-A
Country	US
Kind code	B2
Filing date	Dec 15, 2017
Priority date	Dec 15, 2017
Publication date	Oct 20, 2020
Grant date	Oct 20, 2020

Abstract

Systems and methods for generating answers to questions. One method includes receiving a question having question terms; identifying candidate answers to the question having answer terms; searching data sources to determine passages including either a question term or an answer term in the candidate answer; scoring the passages for candidate answers using a scoring mechanism, the scoring mechanism computing a first degree of relevance of the passage to the question terms, computing a second degree of relevance of the passage to the answer terms of one of the candidate answers, and determining a score for the passage by combining the first degree of relevance and the second degree of relevance; ranking candidate answers to the question based on the scores associated with the scoring each of the passages for each of the candidate answers; and providing an answer to the question based on ranking of the candidate answers.

First claim

What is claimed is:

1. A system for generating answers to questions, the system comprising:
an electronic processor configured to receive a question having a plurality of question terms;
identify a plurality of candidate answers to the question, wherein each of the plurality of candidate answers includes a plurality of answer terms;
for each of the plurality of candidate answers, search a plurality of data sources using a search query including at least one or more of the plurality of question terms and one or more of the plurality of the answer terms included in the candidate answer to determine supporting evidence passages including at least one of the one or more of the plurality of question terms, and one or more of the plurality of the answer terms included in the candidate answer;
score each of the supporting evidence passages for each of the plurality of candidate answers using a scoring mechanism, the scoring mechanism computing a first degree of relevance of the supporting evidence passage to the plurality of question terms, computing a second degree of relevance of the supporting evidence passage to the plurality of answer terms of one of the plurality of candidate answers, and determining a score for the supporting evidence passage by combining the first degree of relevance and the second degree of relevance;
rank the plurality of candidate answers to the question based on the scores associated with the scoring each of the supporting evidence passages for each of the plurality of candidate answers; and
provide an answer to the question based on ranking of the plurality of candidate answers.

2. The system of claim 1 , wherein
one or more of the plurality of question terms and one or more of the plurality of the answer terms included in the candidate answer are tagged as optional.

3. The system of claim 2 , wherein the electronic processor is further configured to
retrieve the plurality of supporting evidence passages from a corpus of data using the search query.

4. The system of claim 3 , wherein the electronic processor is further configured to
iterate through a set of passage scorers for each of the supporting evidence passages, and
score an extent to which each of the passage scorer determines that the supporting evidence passages answers the question in support of an alternate answer, wherein the alternate answer matches the candidate answer.

5. The system of claim 4 , wherein the electronic processor is further configured to
determine a match score for a passage by computing a weight for each question term and dividing a sum of weights of each question term found in the passage by a sum of weights of all the question terms.

6. The system of claim 4 , wherein the electronic processor is further configured to
for each of the set of passage scorers, combine the scores for the supporting evidence passages into one final score indicating that passage scorer's conclusion for the alternate answer.

7. The system of claim 6 , wherein the electronic processor is further configured to
combine the scores for the supporting evidence passages by at least one of summing the scores and taking a maximum value of the scores.

8. The system of claim 1 , wherein the electronic processor is further configured to
compute a first match score for the plurality of question terms,
compute a second match score for the plurality of answer terms, and
combine the first match score and the second match score.

9. The system of claim 8 , wherein the electronic processor is further configured to
combine the first match score and the second match score by multiplying the first match score and the second match score.

10. The system of claim 8 , wherein the electronic processor is further configured to
combine the first match score and the second match score by determining a harmonic mean of the first match score and the second match score.

11. The system of claim 4 , wherein the electronic processor is further configured to
compute a first semantic score of a passage, the first semantic score associated with the question;
compute a second semantic score of the passage, the second semantic score associated with the candidate answer; and
combine the first semantic score and the second semantic score.

12. The system of claim 11 , wherein the electronic processor is further configured to
combine the first semantic score and the second semantic score using cosine as a distance metric.

13. The system of claim 11 , wherein the electronic processor is further configured to
combine the first semantic score and the second semantic score using word mover's distance (WMD) as a distance metric.

14. The system of claim 4 , wherein the scorer uses at least one of a BLEU metric score or a Rouge metric score to compute a similarity of a set of passages to the question and the candidate answer.

15. A method of generating answers to questions, the method comprising:
receiving, with an electronic processor, a question having a plurality of question terms;
identifying, with the electronic processor, a plurality of candidate answers to the question, wherein each of the plurality of candidate answers includes a plurality of answer terms;
for each of the plurality of candidate answers, searching a plurality of data sources using a search query including at least one or more of the plurality of question terms and one or more of the plurality of the answer terms included in the candidate answer to determine supporting evidence passages including at least one of the one or more of the plurality of question terms, and one or more of the plurality of the answer terms included in the candidate answer;
scoring, with the electronic processor, each of the supporting evidence passages for each of the plurality of candidate answers using a scoring mechanism, the scoring mechanism computing a first degree of relevance of the supporting evidence passage to the plurality of question terms, computing a second degree of relevance of the supporting evidence passage to the plurality of answer terms of one of the plurality of candidate answers, and determining a score for the supporting evidence passage by combining the first degree of relevance and the second degree of relevance;
ranking, with the electronic processor, the plurality of candidate answers to the question based on the scores associated with the scoring each of the supporting evidence passages for each of the plurality of candidate answers; and
providing an answer to the question based on ranking of the plurality of candidate answers.

16. The method of claim 15 ,
wherein one or more of the plurality of question terms and one or more of the plurality of the answer terms included in the candidate answer are tagged as optional.

17. The method of claim 16 , further comprising:
determining the first degree of relevance and the second degree of relevance using one or more weightings selected from the group consisting of a lexical proximity of terms, a distributional semantics, and a statistical model.

18. The method of claim 17 , further comprising:
iterating through a set of passage scorers for each of the supporting evidence passages; and
scoring the extent to which each of the passage scorer determines that the supporting evidence passages answers the question in support of an alternate answer, wherein the alternate answer matches the candidate answer.

19. The method of claim 15 , further comprising:
computing a first match score for the plurality of question terms;
computing a seco

Assignees

• IBM

Inventors

• Murdock Iv James William
• Ha Eun
• Hang Chung-Wei
• Hasan Kazi
• Markandaiah Nisarga
• Nolan Christopher Munjal
• Pan Lin
• Peddi Sai Prathyusha
• Swift Mary Diane

Classifications

• G06F16/3329 Primary

  Natural language query formulation · CPC title

• G06F16/24578 Primary

  using ranking · CPC title

• G06F16/3344 Primary

  using natural language analysis · CPC title

• G06F16/24575

  using context · CPC title

• G06F40/30

  Semantic analysis · CPC title