Preserving conceptual distance within unstructured documents

Patent metadata

Field	Value
Publication number	US-9424298-B2
Application number	US-201414508200-A
Country	US
Kind code	B2
Filing date	Oct 7, 2014
Priority date	Oct 7, 2014
Publication date	Aug 23, 2016
Grant date	Aug 23, 2016

Abstract

A method, system and computer-usable medium are disclosed for preserving conceptual distance within unstructured documents by characterizing conceptual relationships. Natural language processing is applied to content in a plurality of documents to identify topics and subjects. Analytic analysis is then applied to the identified topics and subjects to identify concepts. The content in each of the plurality of documents is partitioned into a first structured hierarchy, preserving at least one structure in each document inherent in the each document. Access is then provided to the content through a first index based upon utilizing the first structured hierarchy and through a second index utilizing a second structured hierarchy. The conceptual relationship criteria are based upon a directed graph with weights based upon a similarity and a distance based upon concepts.

First claim

What is claimed is:

1. A system comprising:
a processor;
a data bus coupled to the processor; and
a computer-usable medium embodying computer program code, the computer-usable medium being coupled to the data bus, the computer program code used for characterizing content of documents by conceptual relationships and comprising instructions executable by the processor and configured for:
applying natural language processing (NLP) to content in a plurality of documents to identify topics and subjects;
applying analytic analysis to the topics and subjects to identify a conceptual relationship of the content in the plurality of documents;
partitioning the content in each of the plurality of documents into a first structured hierarchy, preserving at least one structure in each document inherent in the each document; and
providing access to content through a first index based upon utilizing the first structured hierarchy and through a second index utilizing a second structured hierarchy; and wherein
the content is characterized by optimizing a vector space model representation of the documents, the optimization performed by a system capable of answering questions, where:
the content from the plurality of documents is ingested by the system;
natural language processing is applied to the content in the plurality of documents to identify terms, topics, subjects and concepts;
the content is partitioned according to a semantic parse distance to identify a context for partitioned content;
the content and context is represented, by the system, utilizing a vector space model;
entries in the vector space model are eliminated based on a difference criteria; and
an iterative genetic algorithm is applied to optimize features of the vector space model.

2. The system of claim 1 , wherein:
the conceptual relationship is based upon a directed graph with weights based upon a similarity and a distance based upon concepts.

3. The system of claim 1 , wherein:
the distance is based upon a topic hierarchy.

4. The system of claim 1 , wherein:
a ground truth is an optimized feature.

5. The system of claim 4 , wherein:
the genetic algorithm determines which features are used during the ingesting and has weighting based on semantic distance.

6. A non-transitory, computer-readable storage medium embodying computer program code, the computer program code comprising computer executable instructions configured for:
applying natural language processing (NLP) to content in a plurality of documents to identify topics and subjects;
applying analytic analysis to the topics and subjects to identify a conceptual relationship of the content in the plurality of documents;
partitioning the content in each of the plurality of documents into a first structured hierarchy, preserving at least one structure in each document inherent in the each document; and
providing access to content through a first index based upon utilizing the first structured hierarchy and through a second index utilizing a second structured hierarchy; and wherein
the content is characterized by optimizing a vector space model representation of the documents, the optimization performed by a system capable of answering questions, where:
the content from the plurality of documents is ingested by the system;
natural language processing is applied to the content in the plurality of documents to identify terms, topics, subjects and concepts;
the content is partitioned according to a semantic parse distance to identify a context for partitioned content;
the content and context is represented, by the system, utilizing a vector space model;
entries in the vector space model are eliminated based on a difference criteria; and
an iterative genetic algorithm is applied to optimize features of the vector space model.

7. The non-transitory, computer-readable storage medium of claim 6 , wherein:
the conceptual relationship is based upon a directed graph with weights based upon a similarity and a distance based upon concepts.

8. The non-transitory, computer-readable storage medium of claim 6 , wherein:
the distance is based upon a topic hierarchy.

9. The non-transitory, computer-readable storage medium of claim 6 , wherein:
a ground truth is an optimized feature.

10. The non-transitory, computer-readable storage medium of claim 9 , wherein:
the genetic algorithm determines which features are used during the ingesting and has weighting based on semantic distance.

11. The non-transitory, computer-readable storage medium of claim 6 , wherein the computer executable instructions are deployable to a client system from a server system at a remote location.

12. The non-transitory, computer-readable storage medium of claim 6 , wherein the computer executable instructions are provided by a service provider to a user on an on-demand basis.

Assignees

• IBM

Inventors

• Bufe John P
• Winkler Timothy P

Classifications

• G06F40/295

  Named entity recognition · CPC title

• G06F16/93

  Document management systems · CPC title

• G06F16/313

  Selection or weighting of terms for indexing · CPC title

• G06F40/30

  Semantic analysis · CPC title

• G06F16/2272 Primary

  Management thereof · CPC title