Data stream processing

Patent metadata

Field	Value
Publication number	US-9613123-B2
Application number	US-42287509-A
Country	US
Kind code	B2
Filing date	Apr 13, 2009
Priority date	Apr 13, 2009
Publication date	Apr 4, 2017
Grant date	Apr 4, 2017

Abstract

A method of processing a stream of raw data from a plurality of distributed data producing devices includes reducing the raw data to a plurality of representative synopsis coefficients, organizing the synopsis coefficients into a data structure with at least three dimensions, including a time window dimension and an accuracy dimension. Responsive to a detected anomaly in the data structure, at least one of a predetermined autonomous action and an action directed by a user is performed.

First claim

What is claimed is:

1. A method of processing a stream of raw data from a plurality of distributed data producing devices, said method comprising:
reducing said raw data to a plurality of representative synopsis coefficients, each synopsis coefficient being a numeric value representative of two or more different values from said raw data;
organizing said synopsis coefficients into a data structure comprising at least three dimensions, wherein said dimensions comprise a location dimension, a time window dimension and an accuracy dimension, and wherein the location dimension organizes the synopsis coefficients topologically, the time window dimension organizes the synopsis coefficients according to a time when the raw data was received, and the accuracy dimension determines how many of the synopsis coefficients are examined to identify two or more most significant anomalies;
identifying the two or more most significant anomalies in a computational cube associated with the data structure; and
responsive to the identification of the two or more most significant anomalies in the computational cube, performing a remedial action to remedy one or more problems associated with the two or more most significant anomalies.

2. The method of claim 1 , further comprising providing an interactive visual analytics display to a user of at least one of: said stream of raw data, said synopsis coefficients, said data structure, said two or more most significant anomalies, and said predetermined autonomous action.

3. The method of claim 2 , further comprising providing said interactive visual analytics display to said user in accordance with at least one level of abstraction determined by said user.

4. The method of claim 2 , wherein said interactive visual analytics display comprises a correlation map between at least two selected measurements in said raw data.

5. The method of claim 1 , wherein said data producing devices comprise sensors.

6. The method of claim 5 , wherein said sensors are distributed throughout a data center.

7. The method of claim 1 , wherein said synopsis coefficients are calculated by:
forming a binary tree out of said raw data in said stream, said binary tree comprising hierarchical nodes and leaves, wherein said hierarchical nodes correspond to said synopsis coefficients and said leaves correspond to elements of said raw data; and
computing a weighted difference between the average values of the child leaves of each parent node in said binary tree, wherein said computed weighted differences are said synopsis coefficients.

8. The method of claim 1 , wherein said accuracy dimension of said data structure is dependent on a level of abstraction of said synopsis coefficients.

9. The method of claim 1 , wherein said predetermined autonomous action comprises at least one of a notification to a user and a feedback signal sent to a controller.

10. The method of claim 1 , wherein said accuracy dimension determines how many of said synopsis coefficients are used to detect for said anomaly.

11. A system, comprising:
a plurality of distributed data producing devices; and
at least one computing device communicatively coupled to said distributed data producing devices;
wherein said computing device is configured to:
reduce a stream of raw data received from said distributed data producing devices to a plurality of representative synopsis coefficients, wherein reducing a stream of raw data comprises aggregating the synopsis coefficients using one of a count function, an average function, and a standard deviation function;
organize said synopsis coefficients into a data structure comprising at least three dimensions, wherein said dimensions comprise a location dimension, a time window dimension and an accuracy dimension, and wherein the location dimension organizes the synopsis coefficients topologically, the time window dimension organizes the synopsis coefficients according to a time when the raw data was received, and the accuracy dimension determines how many of the synopsis coefficients are examined to identify two or more most significant anomalies; and
perform at least one of a predetermined autonomous response and a user directed action responsive to a detected anomaly in said data structure.

12. The system of claim 11 , wherein said computing device is further configured to provide an interactive visual analytics display to said user, wherein said interactive visual analytics display comprises at least one of: said stream of raw data, said synopsis coefficients, said data structure, said detected anomaly, and said predefined rule.

13. The system of claim 11 , wherein said computing device is further configured to calculate said synopsis coefficients by:
forming a binary tree out of said raw data in said stream, said binary tree comprising hierarchical nodes and leaves, wherein said hierarchical nodes correspond to said synopsis coefficients and said leaves correspond to elements of said raw data; and
computing a weighted difference between the average values of the child leaves of each parent node in said binary tree, wherein said computed weighted differences are said synopsis coefficients.

14. The system of claim 11 , wherein said data producing devices comprise sensors.

15. The system of claim 14 , wherein said sensors are distributed throughout a data center.

16. The system of claim 11 , wherein said predetermined autonomous action comprises at least one of a notification to a user and a feedback signal to a controller.

17. The system of claim 11 , wherein said accuracy dimension determines how many of said synopsis coefficients are used to detect for said anomaly.

18. The system of claim 11 , wherein said computing device detects said anomaly in said data structure using at least one of sequence patterns, semantic correlations or threshold violations.

19. The system of claim 18 , wherein said semantic correlation compares two different event types from different types of sensors among said data producing devices.

20. The system of claim 11 , wherein said at least one computing device comprises a data stream management system wherein continuous queries are authored using a Continuous Query Language and run against said synopsis coefficients in said data structure to detect for anomalies.

21. A computer program product for processing a stream of raw data from a plurality of distributed data producing devices, said computer program product comprising:
a non-transitory computer usable medium having computer usable program code embodied therewith, said computer usable program code comprising:
computer usable program code configured to cause a computer to reduce said raw data to a plurality of representative synopsis coefficients using one of a count function, an average function, and a standard deviation function;
computer usable program code configured to cause said computer to organize said synopsis coefficients into a data structure comprising at least three dimensions, wherein said dimensions comprise a location dimension, a time window dimension and an accuracy dimension, and wherein the location dimension organizes the synopsis coefficients topologically, the time window dimension organizes the synopsis coefficients according to a time when the raw data was received, and the accuracy dimension determines how many of the synopsis coefficients are examined to identify two or more most significant anomalies; and
computer usable program code configured to, responsive to a detected anomaly

Assignees

• Gupta Chetan Kumar
• Wang Song
• Ari Ismail
• Hao Ming C
• Dayal Umeshwar
• Mehta Abhay
• Hewlett Packard Entpr Dev Lp

Inventors

• Gupta Chetan Kumar
• Wang Song
• Ari Ismail
• Hao Ming C
• Dayal Umeshwar
• Mehta Abhay

Classifications

• G06F16/283 Primary

  Multi-dimensional databases or data warehouses, e.g. MOLAP or ROLAP · CPC title

• G06F16/904

  Browsing; Visualisation therefor (for navigating the web G06F16/954; browsing optimisation for the web G06F16/957) · CPC title

• G06F16/35

  Clustering; Classification · CPC title

• G06F17/30994

  Physics · mapped topic

• G06F17/30592 Primary

  Physics · mapped topic