Pluggable fault detection tests for data pipelines

Patent metadata

Field	Value
Publication number	US-9772934-B2
Application number	US-201514877229-A
Country	US
Kind code	B2
Filing date	Oct 7, 2015
Priority date	Sep 14, 2015
Publication date	Sep 26, 2017
Grant date	Sep 26, 2017

Abstract

Discussed herein are embodiments of methods and systems which allow engineers or administrators to create modular plugins which represent the logic for various fault detection tests that can be performed on data pipelines and shared among different software deployments. In some cases, the modular plugins each define a particular test to be executed against data received from the pipeline in addition to one or more configuration points. The configuration points represent configurable arguments, such as variables and/or functions, referenced by the instructions which implement the tests and that can be set according to the specific operation environment of the monitored pipeline.

First claim

The invention claimed is:

1. A method for detecting faults related to a pipeline of a data pipeline system, the method comprising:
a fault detection system receiving a plugin comprising a) one or more instructions representing a test to perform on data processed by the data pipeline system and b) one or more configuration points;
wherein the data pipeline system receives source data from one or more data sources and applies one or more transformations to the source data to produce transformed data before storage of the transformed data in one or more data sinks;
the fault detection system receiving, via a first graphical user interface, one or more settings corresponding to the one or more configuration points;
the fault detection system receiving test data from the data pipeline system, wherein the test data comprises a sample of the transformed data after the one or more transformations;
the fault detection system determining to run the test defined by the plugin on the data pipeline system including executing the one or more instructions of the plugin based on the one or more settings for the one or more configuration points and the test data, wherein a result of executing the one or more instructions includes at least a test result status indicator;
wherein the transformed data comprises tabular data;
wherein the sample comprises a portion of the tabular data;
wherein the test result status indicator is based, at least in part, on the result of executing the one or more instructions including determining:
(a) whether the sample contains a correct number of columns according to a schema for the transformed data,
(b) whether data in each column of the sample adheres to a data type of the column as specified in a schema for the transformed data,
(c) whether data in each column of the sample improperly contains NULL values according to a schema for the transformed data, or
any combination of (a), (b), or (c); and

the fault detection system causing display of a second graphical user interface that visibly presents at least the test result status indicator.

2. The method of claim 1 , wherein determining to run the test is performed based on a configuration point of the one or more configuration points that defines a time interval for periodically executing the test.

3. The method of claim 1 , wherein the test is performed by training a classifier based on a historical sample of the transformed data and, after the classifier has been trained, using the classifier to predict a test result status indicator based on the test data.

4. The method of claim 3 , wherein the classifier is implemented using an artificial neural network.

5. The method of claim 1 , wherein the test result status indicator is one of a plurality of test result status indicators that include at least a test result status representing that a fault occurred with the data pipeline system, a test result status representing that a fault has potentially occurred with the data pipeline system, and a test result status representing that no fault has occurred with the data pipeline system.

6. The method of claim 1 , wherein the data pipeline system includes a plurality of pipelines and the second graphical user interface displays a plurality of test result status indicators, each test result status indicator of the plurality of test result status indicators relating to a plurality of tests performed on a particular pipeline during a particular time period.

7. The method of claim 6 , wherein each test result status indicator of the plurality of test result status indicators is generated by using a worst test result status indicator among test result status indicators for the plurality of tests performed on the particular pipeline during the particular time period.

8. The method of claim 7 , wherein each particular test result status indicator of the plurality of test result status indicators is displayed as or in relation to a widget which, when selected, causes display of a third graphical user interface that presents the plurality of tests for the particular pipeline during the particular time period.

9. The method of claim 8 , wherein each particular test of the plurality of tests is displayed in the third graphical user interface as or in relation to a widget which, when selected, causes display of a fourth graphical user interface that presents detailed information for the particular test.

10. The method of claim 9 , wherein the detailed information for the particular test is displayed in relation to a widget which, when selected, causes a test result status indicator of the particular test to be treated as though no fault was detected.

11. The method of claim 1 , wherein the one or more configuration points include one or more of: variables referenced by the one or more instructions or functions referenced by the one or more instructions.

12. The method of claim 1 , wherein the one or more instructions perform the test by inspecting log data generated by the data pipeline system for one or more results of the data pipeline system executing one or more checks for faults involving the data pipeline system.

13. The method of claim 1 , wherein the second graphical user interface is displayed via a client application.

14. The method of claim 13 , wherein the fault detection system receives the plugin via the client application.

15. The method of claim 1 , wherein the test data comprises a sample of the source data before the one or more transformations.

16. The method of claim 15 , wherein the one or more configuration points specify collection of the sample of the source data from the one or more data sources.

17. The method of claim 15 , wherein the test result status indicator is based, at least in part, on the result of executing the one or more instructions including determining:
(a) whether the sample of the source data contains a correct number of columns according to a schema for the source data,
(b) whether data in each column of the sample of the source data adheres to a data type of the column as specified in a schema for the source data,
(c) whether data in each column of the sample of the source data improperly contains NULL values according to a schema for the source data, or
any combination of (a), (b), or (c).

18. The method of claim 1 , wherein the one or more configuration points specify collection of the sample of the transformed data from the one or more data sinks.

19. A fault detection system for detecting faults related to a pipeline of a data pipeline system, the fault detection system comprising:
one or more computing devices having one or more processors and memory;
a client communication interface configured to:
receive a plugin comprising one or more instructions representing a test to perform on the data pipeline system and one or more configuration points, wherein the data pipeline system is configured to receive source data from one or more data sources and configured to apply one or more transformations to the source data to produce transformed data before storage of the transformed data in one or more data sinks;
and
receive one or more settings corresponding to the one or more configuration points via a first graphical user interface;

a pipeline communication interface configured to receive test data from the data pipeline system, wherein the test data comprises a metric reflecting an amount of the transformed data after the one or more transformations;
a data analysis subsystem configured to run the test defined by the plug

Assignees

• Palantir Technologies Inc
• Palantir Technologies Inc

Inventors

• Maag Peter
• Albertson Jacob
• Newman Jared
• Lynch Matthew
• Albin Maciej
• Nordling Viktor

Classifications

• G06F11/3692 Primary

  for test results analysis · CPC title

• G06F11/0775 Primary

  Content or structure details of the error report, e.g. specific table structure, specific error fields · CPC title

• G06F11/0751

  Error or fault detection not based on redundancy (power supply failures G06F1/30; network fault management H04L41/06) · CPC title

• G06F11/3684

  for test design, e.g. generating new test cases · CPC title

• G06F11/3688

  for test execution, e.g. scheduling of test suites · CPC title