Text indexing and passage retrieval

Patent metadata

Field	Value
Publication number	US-10430445-B2
Application number	US-201414485176-A
Country	US
Kind code	B2
Filing date	Sep 12, 2014
Priority date	Sep 12, 2014
Publication date	Oct 1, 2019
Grant date	Oct 1, 2019

Abstract

Methods and systems for indexing document passages are presented. In some embodiments, a computing device may identify a plurality of documents that comprise a plurality of passages. A passage index comprising a plurality of entries may be generated. Each entry may comprise keywords from a passage of the plurality of passages in one of the plurality of documents. Each entry may further comprise at least one annotation associated with the passage. A search query comprising at least one search keyword may be received. The passage index for each document of the plurality of documents may be analyzed using the at least one search keyword to identify at least one passage from the plurality of documents that matches the search query. In response to the query, the at least one passage may be presented.

First claim

What is claimed is:

1. A method for indexing document passages, the method comprising:
identifying, by a computing device, a plurality of documents, wherein each document of the plurality of documents comprises a plurality of passages;
generating, by the computing device and by parsing each document of the plurality of documents according to a predefined document structure parsing algorithm, a passage index comprising a plurality of entries, each entry comprising: a keyword from a passage of the plurality of passages in one of the plurality of documents, at least one annotation associated with the passage, and an association between the keyword from the passage and the at least one annotation;
determining one or more intent identifiers corresponding to a question input, wherein the question input comprises a natural language input received by the computing device;
determining a search query comprising at least one search keyword corresponding to an intent identifier of the one or more intent identifiers;
assigning the one or more intent identifiers to at least one passage from the plurality of documents in the passage index;
determining which intent identifiers of the one or more intent identifiers best match the at least one search keyword in the passage index based on, at least in part, natural language processing;
using the at least one search keyword to identify the at least one passage from the plurality of documents that matches the search query;
scanning for one or more boundary markers, determined via the predefined document structure parsing algorithm and comprising the at least one annotation associated with the at least one passage, to identify a start and an end of the at least one passage;
and generating, by the computing device, and using the at least one passage, a natural language response output comprising an answer to the question input, wherein generating the natural language response output causes the answer to the question input to be presented.

2. The method of claim 1 , wherein annotations in the passage index comprise text from at least one of a document title of a document corresponding to the keyword, a section heading corresponding to the keyword, context from one of the plurality of passages located proximal to the section heading, concept identifiers related to the keyword, a synonym of the keyword, a part of speech for the keyword, syntactic dependencies of the keyword, and one or more boundary markers in the document near the keyword.

3. The method of claim 1 , further comprising:
searching, by the computing device, portions of the passage index for matches between an entry in the passage index and the at least one search keyword; and
scoring, by the computing device, each passage of each of the plurality of documents based on a number of matches between passage index entries for a given passage and the at least one search keyword.

4. The method of claim 1 , wherein the passage index comprises passage text from the passage and wherein the at least one annotation is added to the passage index at an end of a portion of the passage text.

5. The method of claim 1 , wherein the at least one annotation comprises text from a document located at a non-proximal location in the document relative to the keyword.

6. The method of claim 1 , wherein one or more documents of the plurality of documents comprises well-formed sentences, the method further comprising:
dividing, by the computing device, each of the one or more documents comprising well-formed sentences, to generate a plurality of sentence oriented documents; and
associating, by the computing device, keywords in one of the plurality of sentence oriented documents with annotations comprising information related to the keywords.

7. The method of claim 1 , further comprising determining the one or more boundary markers by:
identifying, while parsing each document of the plurality of documents according to the predefined document structure parsing algorithm, keywords located at least one of: a beginning of the passage, an end of the passage, a beginning of a sentence, or the end of a sentence;
generating the at least one annotation associated with the passage, wherein the at least one annotation indicates that the keywords comprise the one or more boundary markers; and
storing the at least one annotation in the passage index.

8. The method of claim 1 , further comprising determining synonyms and concept identifiers corresponding to the search query.

9. The method of claim 1 , wherein the intent identifiers correspond to words other than words comprising the question input.

10. The method of claim 1 , wherein generating the natural language response output comprises identifying, in the at least one passage, an answer to the question input by processing one or more of syntactic dependencies, anaphoric relations, semantic relations, and antecedents in portions of the at least one passage.

11. The method of claim 1 , further comprising converting the question input into a logical operation form prior to determining the one or more intent identifiers corresponding to the question input.

12. An apparatus, comprising:
at least one processor; and
at least one memory storing computer-readable instructions that, when executed by the at least one processor, cause the apparatus to:
identify a plurality of documents, wherein each document of the plurality of documents comprises a plurality of passages;
generate, by parsing each document of the plurality of documents according to a predefined document structure parsing algorithm, a passage index comprising a plurality of entries, each entry comprising: a keyword from a passage of the plurality of passages in one of the plurality of documents, at least one annotation associated with the passage, and an association between the keyword from the passage and the at least one annotation;
determine one or more intent identifiers corresponding to a question input, wherein the question input comprises a natural language input received by the apparatus;
determine a search query comprising at least one search keyword corresponding to an intent identifier of the one or more intent identifiers;
assign the one or more intent identifiers to at least one passage from the plurality of documents in the passage index;
determine which intent identifiers of the one or more intent identifiers best match the at least one search keyword in the passage index based on, at least in part, natural language processing;
use the at least one search keyword to identify the at least one passage from the plurality of documents that matches the search query;
scan for one or more boundary markers, determined via the predefined document structure parsing algorithm and comprising the at least one annotation associated with the at least one passage, to identify a start and an end of the at least one passage;
and generate, using the at least one passage, a natural language response output comprising an answer to the question input, wherein generating the natural language response output causes the answer to the question input to be presented.

13. The apparatus of claim 12 , wherein annotations in the passage index comprise text from at least one of a document title of a document corresponding to the keyword, a section heading corresponding to the keyword, context from one of the plurality of passages located proximal to the section heading, concept identifiers related to the keyword, a synonym of the keyword, a part of speech for the keyword, syntactic dependencies of the keyword, and one or more boundary markers in the document near the keyword.

Assignees

• Nuance Communications Inc

Inventors

• Crouch Richard S
• Boston Marisa F
• Ozcan Ali Erdem
• Stubley Peter R

Classifications

• G06F16/313 Primary

  Selection or weighting of terms for indexing · CPC title