Performance and scalability in an intelligent data operating layer system

Patent metadata

Field	Value
Publication number	US-9323767-B2
Application number	US-201213632825-A
Country	US
Kind code	B2
Filing date	Oct 1, 2012
Priority date	Oct 1, 2012
Publication date	Apr 26, 2016
Grant date	Apr 26, 2016

Abstract

Systems and methods that allow for an intelligence platform for distributed processing of big data sets including both structured and unstructured data types across two or more intelligent data operation engine servers. The intelligent data operation engine servers can form a conceptual understanding of content in each electronic file and then cooperates with a distributed index handler to index the conceptual understanding of the electronic file. A query pipeline and the distributed index handler in the intelligence platform cooperate with the two or more intelligent data operation engine servers to improve scalability and performance on the big data sets containing both structured and un-structured electronic files represented in the common index.

First claim

What is claimed is:

1. A system comprising:
an index handler;
data servers configured for distributed processing of structured and unstructured data,
where a first of the data servers is configured to:
use a statistical algorithm to assign weights to terms found in content of electronic files,
use idea distancing between similar concepts to find different words in the electronic files that describe a same idea,
form a conceptual understanding of content in each of the electronic files using the statistical algorithm and the idea distancing, and
cooperate with the index handler to form a common index of the conceptual understanding of each of the electronic files, where the common index includes at least a conceptual understanding of a first of the electronic files that contains structured data and a conceptual understanding of a second of the electronic files that contains unstructured data, and

a query pipeline that includes an action handler that is configured to support a distribution of command actions in a common protocol to the data servers,
wherein the data servers are configurable to selectively run in one of a mirror mode and a non-mirror mode, wherein the data servers in the mirror mode have a same configuration and contain same data, and wherein the data servers in the non-mirror mode are configured differently and contain different data, and
wherein the index handler is configurable to selectively run in one of the mirror mode and the non-mirror mode, and wherein the index handler in the mirror mode is configured to distribute same index data of the common index to the data servers, and the index handler in the non-mirror mode is configured to distribute different portions of the common index to the corresponding data servers.

2. The system of claim 1 , wherein a first instance of the action handler is implemented in a distribution server configured to support and convey distribution of the command actions in the common protocol both to and between the data servers.

3. The system of claim 1 , wherein the first data server is configured to use the idea distancing to form clusters of the electronic files, each of the clusters conveying a similar concept based on the formed conceptual understanding of the electronic files in the respective cluster fitting into a respective category.

4. The system of claim 1 , wherein the index handler is configured to split and index data into the data servers, optimize performance by batching data for each data server to process, and replicate index commands between the data servers.

5. The system of claim 1 , wherein the query pipeline is configured to analyze a statistically most rare occurrence of a term present in search terms of the query, and to determine, based on the analyzing, a most relevant portion of structured data and unstructured data to begin a for the query.

6. The system of claim 1 , wherein the query pipeline is configured to focus a majority of total processing power of the data servers to find electronic files most relevant towards a term in the query that has a most rare occurrence, and wherein the query pipeline is configured to start a search in response to the query that is weighted more heavily on the term having the most rare occurrence.

7. The system of claim 1 , further comprising:
repositories storing the electronic files that are organized by subject matter, and wherein the repositories include both repositories for the structured data and repositories for the unstructured data, and wherein the conceptual understandings are represented in an XML format.

8. The system of claim 1 , wherein the first data server includes an executable dynamic reasoning engine that is configured to interact with one or more processors implementing multi-threaded processes of the first data server to generate the conceptual understandings of the electronic files, and execute queries on the electronic files.

9. The system of claim 1 , wherein conceptual understandings of the electronic files are indexed and organized within the common index, and each conceptual understanding within the common index has one or more pointers to the respective electronic file.

10. The system of claim 1 , wherein the action handler is further configured to propagate query actions to the data servers to search the common index, where the action handler is further configured to:
use the data servers as a pool of servers,
automatically select a primary data server in the pool, and
switch to a secondary data server in the pool when the primary data server fails.

11. The system of claim 1 , wherein the system increases performance of both the index handler and the query pipeline by using database statistics and life cycle management to determine relevant electronic files and to place the relevant electronic files into respective categories for indexing.

12. The system of claim 11 , wherein the first data server is configured to make a determination of which electronic files are most relevant by considering an amount of relevant documents returning to key terms of a query, a strength rating or percentage relevance of those returned documents to the query, and tracked historic data of a most relevant indexed categories to search for previous similar queries.

13. The system of claim 1 , where the action handler is configured to distribute the command actions when a lack of feedback from a particular data server occurs to ensure uninterrupted service when any of the data servers should fail.

14. The system of claim 1 , wherein the first data server is configured to perform adaptive probabilistic concept caching in which frequently-used concepts are maintained in a memory of the first data server and query results are returned using the frequently-used concepts maintained in the memory.

15. The system of claim 1 , wherein the first data server is configured to form the conceptual understanding of a given electronic file by:
eliminating inconsequential information from a content in the given electronic file;
generating a set of key terms, wherein the set of key terms include one or more of singular terms, higher order terms, noun phrases, proper names, and any combination of these; and
assigning a frequency of occurrence weight to each of the key terms.

16. A method comprising:
assigning weights to terms found in content of electronic files;
using idea distancing between similar concepts to find different words in the electronic files that describe a same idea;
using both the assigned weights and the idea distancing to form a conceptual understanding of the content in each of the electronic files;
cooperating with an index handler to form a common index of the conceptual understanding of each of the electronic files, where the common index includes at least a conceptual understanding of a first of the electronic files that contains structured data and a conceptual understanding of a second of the electronic files that contains unstructured data;
distributing, by a query pipeline, command actions in a common protocol to data servers;
selectively configuring the data servers to selectively run in one of a mirror mode and a non-mirror mode, wherein the data servers in the mirror mode have a same configuration and contain same data, and wherein the data servers in the non-mirror mode are configured differently and contain different data; and
selectively configuring the index handler to selectively run in one of the mirror mode and the non-mirror mode, wherein the index handler in the mirror mode is configured to distribute same index data of the common index t

Assignees

• Longsand Ltd

Inventors

• Blanchflower Sean Mark
• Gallagher Darren John

Classifications

• G06F17/30011

  Physics · mapped topic

• G06F17/30094 Primary

  Physics · mapped topic

• G06F16/93

  Document management systems · CPC title

• G06F16/182 Primary

  Distributed file systems · CPC title

• G06F16/134 Primary

  Distributed indices · CPC title