System for rapid interactive exploration of big data

Patent metadata

Field	Value
Publication number	US-11429623-B2
Application number	US-202016738999-A
Country	US
Kind code	B2
Filing date	Jan 9, 2020
Priority date	Jan 9, 2020
Publication date	Aug 30, 2022
Grant date	Aug 30, 2022

Abstract

An apparatus for estimating analytics and interactive exploration of big data, stored and/or streaming, using approximate query processing is presented. The apparatus comprises a model constructor and a sampler. The model constructor identifies important predictors variables in big data using feature selection, predictor variables, and outcome variables and partitions the important predictor variables into one or more stratifications based either the identified interactions or identified relationships. The sampler generates a subset of data by querying the big data using a query constructed based on at least one stratification. The subset of data can be fed into an analytics generator. The analytics generator generates analytics data for the outcome variables based on the subset of data and an analytics algorithm and a visualization, e.g. an interactive visualization, comprising the outcome variables, the important predictor variables, the stratification, the subset of data, and the analytics data.

First claim

What is claimed is:

1. A system for estimating analytics and interactive exploration of big data, the system comprising:
a model constructor configured to:
identify in the big data predictors variables using feature selection, at least one predictor variable, and at least one outcome variable;
identify in the big data at least one of interactions between the predictor variables and relationships of structure in the predictor variables using an incremental machine learning algorithm until a desired level of structure has been identified based on a user-defined criteria, wherein the incremental machine learning is used to update the predictor variables based on newly obtained data without a need to reprocess over all previously processed data, wherein the model constructor is further configured to identify a contribution of one predictor variable of the predictor variables as modified by one or more other predictor variables of the predictor variables to identify an interaction of the at least one interactions between the predicator variables, wherein a combined contribution of all predicator variables involved in the interaction is greater than a simple sum over individual contributions of each of the predictor variables;
interaction means that the contribution of one predictor variable is modified by one or many other predictor variables, so that the combined contribution of all variables involved in the interaction is greater than the simple sum over the individual contributions attributable to each variable;
partition the predictor variables into at least one stratification based on at least one of the identified interactions and identified relationships; and

a sampler configured to:
generate a subset of data by querying the big data using a query constructed based on the at least one stratification;

an analytics generator configured to:
generate analytics data for the at least one outcome variable based on the subset of data and at least one analytics algorithm; and
generate at least one visualization comprising at least one of the at least one outcome variable, the predictor variables, the at least one stratification, the subset of data, and the analytics data;

wherein the big data is at least one of stored data and streaming data.

2. The system of claim 1 , wherein the at least one predictor variable and the at least one outcome variable are update variables and the subset of data is an updated subset of data.

3. The system of claim 1 , wherein the predictor variables are identified and partitioned using at least one tree algorithm; wherein the query is constructed using approximate query processing.

4. The system of claim 1 , wherein the predictor variables and outcome variables are at least one of continuous and categorical variables of interest selected from logical data columns.

5. The system of claim 1 , wherein the sampler is further configured to generate the subset of data by querying the big data using a query constructed based on the at least one stratification according to an accuracy criteria.

6. The system of claim 5 , wherein the accuracy criteria is one of time and an acceptable error limit.

7. An apparatus for estimating analytics and interactive exploration of big data, the apparatus comprising:
a model constructor configured to:
continuously identify in the big data at least one of interactions between the predictor variables and relationships of structure in the predictor variables using an incremental machine learning algorithm until a desired level of structure has been identified based on a user-defined criteria, wherein the incremental machine learning is used to update the predictor variables based on newly obtained data without a need to reprocess over all previously processed data,
wherein the model constructor is further configured to identify a contribution of one predictor variable of the predictor variables as modified by one or more other predictor variables of the predictor variables to identify an interaction of the at least one interactions between the predicator variables, wherein a combined contribution of all predicator variables involved in the interaction is greater than a simple sum over individual contributions of each of the predictor variables;
partition, continuously, predictor variables into at least one stratification based on at least one of the identified interactions and identified relationships; and

a sampler configured to:
continuously generate a subset of data by querying the big data using a query constructed based on the at least one stratification;

wherein the big data is at least one of stored data and streaming data.

8. The apparatus of claim 7 , wherein the model constructor is further configured to:
identify in the big data predictors variables using feature selection, at least one predictor variable, and at least one outcome variable.

9. The apparatus of claim 7 , further comprising:
an analytics generator configured to:
generate analytics data for the at least one outcome variable based on the subset of data and at least one analytics algorithm; and
generate at least one visualization comprising at least one of the at least one outcome variable, the predictor variables, the at least one stratification, the subset of data, and the analytics data.

10. The apparatus of claim 7 , wherein the at least one predictor variable and the at least one outcome variable are update variables and the subset of data is an updated subset of data.

11. The apparatus of claim 7 , wherein the predictor variables are identified and partitioned using at least one tree algorithm; wherein the query is constructed using approximate query processing.

12. The apparatus of claim 7 , wherein the predictor variables and outcome variables are at least one of continuous and categorical variables of interest selected from logical data columns.

13. The apparatus of claim 7 , wherein the sampler is further configured to generate the subset of data by querying the big data using a query constructed based on the at least one stratification according to an accuracy criteria.

14. The system of claim 12 , wherein the accuracy criteria is one of time and an acceptable error limit.

15. A method for estimating analytics and interactive exploration of big data, the method comprising:
identify in the big data predictors variables using feature selection, at least one predictor variable, and at least one outcome variable;
identify in the big data at least one of interactions between the predictor variables and relationships of structure in the predictor variables using an incremental machine learning algorithm until a desired level of structure has been identified based on a user-defined criteria, wherein the incremental machine learning is used to update the predictor variables based on newly obtained data without a need to reprocess over all previously processed data, wherein identifying an interaction of the at least one interactions comprises identifying a contribution of one predictor variable of the predictor variables as modified by one or more other predictor variables of the predictor variables, wherein a combined contribution of all predicator variables involved in the interaction is greater than a simple sum over individual contributions of each of the predictor variables;
partition the predictor variables into at least one stratification based on at least one of the identified interactions and identified relationships;
generate a subset of data by querying the big data using a query constructed based on the at least one stratification;
generate analytics da

Assignees

• Tibco Software Inc

Inventors

• Hill Thomas
• Katz David
• O'Connell Michael
• Ramnarayan Jags
• Rope Daniel J

Classifications

• G06F16/248

  Presentation of query results · CPC title

• G06F16/2465 Primary

  Query processing support for facilitating data mining operations in structured databases · CPC title

• G06F16/2462 Primary

  Approximate or statistical queries · CPC title