Parallelizing semantically split documents for processing

Patent metadata

Field	Value
Publication number	US-9971761-B2
Application number	US-201514734404-A
Country	US
Kind code	B2
Filing date	Jun 9, 2015
Priority date	Dec 22, 2014
Publication date	May 15, 2018
Grant date	May 15, 2018

Abstract

In an approach for parallelizing document processing in an information handling system, a processor receives a document, wherein the document includes text content. A processor extracts information from the text content, utilizing natural language processing and semantic analysis, to form tokenized semantic partitions, comprising a plurality of sub-documents. A processor schedules a plurality of concurrently executing threads to process the plurality of sub-documents.

First claim

What is claimed is:

1. A method for parallelizing document processing in an information handling system, the method comprising:
receiving, by one or more processors, a document, wherein the document includes text content, given a particular granularity scope;
extracting, by one or more processors, information from the text content, utilizing natural language processing and semantic analysis, to form tokenized semantic partitions, comprising a plurality of sub-documents, wherein:
the tokenized semantic partitions each have a particular data type;
the plurality of sub-documents are annotated to represent an order of occurrence within the document; and
the annotated plurality of sub-documents allows reconstruction to the particular granularity scope at any point during the extraction;

reconstructing, by one or more processors, the document by scheduling a process for the annotated plurality of sub-documents, wherein:
the scheduling drives each of the annotated sub-documents in parallel by using a memory barrier to enforce an ordering constraint on the annotated plurality of sub-documents based on a data dependent scheduling order using the data types of the sub-documents and a type dependency flow graph for the annotated sub-documents given the particular granularity scope, wherein non-dependent parts of the sub-documents can be processed out-of-order,
the dependency flow graph comprises information about which data types are dependent on other data types in a dependency order, and
by using the type dependency flow graph, the sub-documents that have data types that do not depend upon each other can be driven in parallel and processed out of order of occurrence, while the sub-documents that have data types that depend on each other are constrained and processed according to the dependency order using the memory barrier; and

retrieving, by one or more processors, numbered annotation data within the scheduled and annotated plurality of sub-documents, representing the order of occurrence, wherein the reconstructed document preserves the order of occurrence of previous extractions based on the retrieved numbered annotation data.

2. The method of claim 1 , wherein the plurality of sub-documents are separate components of a document.

3. The method of claim 1 , wherein the process is a plurality of concurrently executing threads.

4. The method of claim 1 , wherein each sub-document is processed using a data dependency workflow, containing annotator metadata, wherein each annotator metadata has a description of input types needed and output types produced.

5. The method of claim 1 , further comprising:
annotating, by one or more processors, each sub-document;
storing, by one or more processors, each annotated sub-document; and
reconstructing, by one or more processors, the document using each sub document, based on information in the annotated sub-document.

6. The method of claim 1 , wherein the plurality of sub-documents are partitioned based on data type and scope of the text content.

7. The method of claim 6 , wherein scope of the text content is selected from a group consisting of word, sentence, and paragraph.

Assignees

• IBM

Inventors

• Desai Rajesh M
• Parker Philip E
• Raphael Roger C
• Taylor Paul S

Classifications

• G06F40/30 Primary

  Semantic analysis · CPC title

• G06F40/284 Primary

  Lexical analysis, e.g. tokenisation or collocates · CPC title

• G06F17/2785

  Physics · mapped topic

• G06F17/277 Primary

  Physics · mapped topic