Ranking answers in ground truth of a question-answering system

Patent metadata

Field	Value
Publication number	US-2017192976-A1
Application number	US-201614988780-A
Country	US
Kind code	A1
Filing date	Jan 6, 2016
Priority date	Jan 6, 2016
Publication date	Jul 6, 2017
Grant date	—

Abstract

A computer-implemented method for ranking answers of question-answer pairs in ground truth. The computer-implemented method includes annotating, via an annotation tool, the domain dictionary, including, at least one or more of lexicons, glossaries, knowledge base query for ranking the question-answer pairs. The computer-implemented method further includes defining search patterns of the annotated domain dictionary. The computer-implemented method further includes. The computer-implemented method further searching a documents corpus or knowledge based query of the domain dictionary for candidate answers of the question-answer pairs, based on the defined search patterns, the candidate are returned based on the search. The computer-implemented method for evaluating the candidate answers with the annotation tool to verify correctness or acceptability of the returned candidate answers based on the search.

First claim

What is claimed is:

1 . A computer-implemented method for ranking answers of question-answer pairs in ground truth, the computer-implemented method comprising:
annotating, via an annotation tool, a domain dictionary, including, at least one or more of lexicons, glossaries, knowledge base query for ranking the question-answer pairs; defining search patterns of the annotated domain dictionary; searching a documents corpus or knowledge based query using domain dictionary for candidate answers of the question-answer pairs, based on the defined search patterns, the candidate are returned based on the search; and evaluating the candidate answers with the annotation tool to verify correctness or acceptability of the returned candidate answers based on the search, if the answers are acceptable, the annotation tool assigns an acceptable ranking to the candidate answers based on a scale.

2 . The computer-implemented method of claim 1 , wherein acceptability features are automatically extracted from ranked acceptable answers.

3 . The computer-implemented method of claim 2 , wherein the features includes at least one of passage length of the answers and sentence structure of the answer.

4 . The computer-implemented method of claim 3 , wherein ranking patterns are generated based on acceptability and ranking.

5 . The computer-implemented method of claim 4 , wherein the generated ranked patterns are presented to an annotator of the annotator tool for review.

6 . The computer-implemented method of claim 5 , wherein the annotator utilizes the generated ranked patterns to augment existing search patterns via the annotator tool.

7 . The computer-implemented method of claim 1 , wherein if the acceptable candidate answers are not returned, the annotator tool augments or refines the domain terms and search patterns and re-searches the document corpus.

8 . A computer system for ranking answers of question-answer pairs in ground truth, the computer-implemented method comprising:
one or more processors, one or more computer-readable memories, one or more computer-readable tangible storage devices and program instructions which are stored on at least one of the one or more storage devices for execution by at least one of the one or more processors via at least one of the one or more memories, the program instructions comprising:
program instructions to annotate, via an annotation tool, a domain dictionary, including, at least one or more of lexicons, glossaries, knowledge base query for ranking the question-answer pairs;
programs instructions to define search patterns of the annotated domain dictionary;
program instructions to search a documents corpus or knowledge based query using the domain dictionary for candidate answers of the question-answer pairs, based on the defined search patterns, the candidate are returned based on the search; and
program instructions to evaluate the candidate answers with the annotation tool to verify correctness or acceptability of the returned candidate answers based on the search, if the answers are acceptable, the annotation tool assigns an acceptable ranking to the candidate answers based on a scale.

9 . The computer system of claim 8 , wherein acceptability features are automatically extracted from ranked acceptable answers.

10 . The computer system of claim 9 , wherein the features includes at least one of passage length of the answers and sentence structure of the answer.

11 . The computer system of claim 10 , wherein ranking patterns are generated based on acceptability and ranking.

12 . The computer system of claim 11 , wherein the generated ranked patterns are presented to an annotator of the annotator tool for review.

13 . The computer system of claim 12 , wherein the annotator utilizes the generated ranked patterns to augment existing search patterns via the annotator tool.

14 . The computer system of claim 8 , wherein if the acceptable candidate answers are not returned, the annotator tool augments or refines the domain terms and search patterns and re-searches the document corpus.

15 . A computer program product for ranking answers of question-answer pairs in ground truth, the computer program-product comprising:
one or more computer-readable tangible storage devices and program instructions stored on at least one of the one or more storage devices, the program instructions comprising:
program instructions to annotate, via an annotation tool, a domain dictionary, including, at least one or more of lexicons, glossaries, knowledge base query for ranking the question-answer pairs;
program instructions to define search patterns of the annotated domain dictionary;
program instructions to search a documents corpus or knowledge based query using the domain dictionary for candidate answers of the question-answer pairs, based on the defined search patterns, the candidate are returned based on the search; and
program instructions to evaluate the candidate answers with the annotation tool to verify correctness or acceptability of the returned candidate answers based on the search, if the answers are acceptable, the annotation tool assigns an acceptable ranking to the candidate answers based on a scale.

16 . The computer program product of claim 15 , wherein acceptability features are automatically extracted from ranked acceptable answers.

17 . The computer program product of claim 16 , wherein the features includes at least one of passage length of the answers and sentence structure of the answer.

18 . The computer program product of claim 17 , wherein ranking patterns are generated based on acceptability and ranking.

19 . The computer program product of claim 18 , wherein the generated ranked patterns are presented to an annotator of the annotator tool for review.

20 . The computer program product of claim 15 , wherein if the acceptable candidate answers are not returned, the annotator tool augments or refines the domain terms and search patterns and re-searches the document corpus.

Assignees

• IBM

Inventors

• Bhatia Dimple
• Brown Armageddon R
• Byron Donna K
• Swift Mary D
• Taieb David D

Classifications

• G06F16/24578 Primary

  using ranking · CPC title

• G06F16/93

  Document management systems · CPC title

• G06F16/24522

  Translation of natural language queries to structured queries · CPC title

• G06F17/3043

  Physics · mapped topic

• G06F17/3053 Primary

  Physics · mapped topic