Data pipeline monitoring

Patent metadata

Field	Value
Publication number	US-10318398-B2
Application number	US-201715498258-A
Country	US
Kind code	B2
Filing date	Apr 26, 2017
Priority date	Jun 10, 2016
Publication date	Jun 11, 2019
Grant date	Jun 11, 2019

Abstract

A method and system for data pipeline monitoring receives an event data object and a current status data object from one or more subsystems of a pipeline. The system analyzes the event data object and the current status data object to determine a first and second validation value. The system, in response to determining that either the first or second validation value is not valid, sends a notification.

First claim

The invention claimed is:

1. A method providing an improvement in monitoring pipelined data transformation processes in distributed computer systems, the method comprising:
receiving an event data object that indicates a historical status of one or more subsystems of a pipeline;
receiving a current status data object that indicates a current status of one or more subsystems of a pipeline;
analyzing the event data object to determine a first validation value of the event data object by applying a first validation criteria to the event data object;
analyzing the current status data object to determine a second validation value of the current status data object by applying a second validation criteria to the current status data object;
each of the first validation criteria and the second validation criteria comprising any of a programming instruction, script, query, configuration file, or business logic that validates the event data object or the current status data object to determine values that represents characteristics of a health state of the pipeline;
based on determining that the first validation value is not valid, causing to be displayed in a graphical user interface the first validation value;
based on determining that the second validation value is not valid, causing to be displayed in the graphical user interface the second validation value;
wherein the method is performed using one or more processors.

2. The method of claim 1 , further comprising:
causing to be displayed in the graphical user interface a summary card for the one or more subsystems that summarizes the first validation value and the second validation value.

3. The method of claim 2 , wherein the summary card for the one or more subsystems further comprises a plurality of numerical values, wherein each numerical value indicates a number of validation values associated with a particular status type for the one or more subsystems.

4. The method of claim 2 , further comprising:
causing to be displayed in the summary card a graph of one or more prior validation results for the one or more subsystems.

5. The method of claim 4 , wherein the graph comprises a line graph.

6. The method of claim 4 , wherein the graph comprises a histogram.

7. The method of claim 1 , wherein a source of the event data object is an event log.

8. The method of claim 1 , wherein the current status data object comprises information regarding one or more of: disk space, memory usage, and central processing unit (CPU) usage.

9. A system providing an improvement in monitoring pipelined data transformation processes in distributed computer systems, the system comprising:
one or more processors;
one or more non-transitory computer-readable storage media storing sequences of instructions which, when executed by the one or more processors, cause the one or more processors to perform:
receiving an event data object that indicates a historical status of one or more subsystems of a pipeline;
receiving a current status data object that indicates a current status of one or more subsystems of a pipeline;
analyzing the event data object to determine a first validation value of the event data object by applying a first validation criteria to the event data object;
analyzing the current status data object to determine a second validation value of the current status data object by applying a second validation criteria to the current status data object;
each of the first validation criteria and the second validation criteria comprising any of a programming instruction, script, query, configuration file, or business logic that validates the event data object or the current status data object to determine values that represents characteristics of a health state of the pipeline;
in response to determining that the first validation value is not valid, causing to be displayed in a graphical user interface the first validation value; and
in response to determining that the second validation value is not valid, causing to be displayed in the graphical user interface the second validation value.

10. The system of claim 9 , further comprising sequences of instructions which, when executed by the one or more processors, cause the one or more processors to perform:
causing to be displayed in the graphical user interface a summary card for the one or more subsystems that summarizes the first validation value and the second validation value.

11. The system of claim 10 , wherein the summary card for the one or more subsystems further comprises a plurality of numerical values, wherein each numerical value indicates a number of validation values associated with a particular status type for the one or more subsystems.

12. The system of claim 10 , further comprising sequences of instructions which, when executed by the one or more processors, cause the one or more processors to perform:
causing to be displayed in the summary card a graph of one or more prior validation results for the one or more subsystems.

13. The system of claim 12 , wherein the graph comprises a line graph.

14. The system of claim 12 , wherein the graph comprises a histogram.

15. The system of claim 9 , wherein a source of the event data object is an event log.

16. The system of claim 9 , wherein the current status data object comprises information regarding one or more of: disk space, memory usage, and central processing unit (CPU) usage.

17. One or more non-transitory computer-readable media storing one or more sequences of instructions, providing an improvement in monitoring pipelined data transformation processes in distributed computer systems and which, when executed by one or more processors, cause:
receiving an event data object that indicates a historical status of one or more subsystems of a pipeline;
receiving a current status data object that indicates a current status of one or more subsystems of a pipeline;
analyzing the event data object to determine a first validation value of the event data object by applying a first validation criteria to the event data object;
analyzing the current status data object to determine a second validation value of the current status data object by applying a second validation criteria to the current status data object;
each of the first validation criteria and the second validation criteria comprising any of a programming instruction, script, query, configuration file, or business logic that validates the event data object or the current status data object to determine values that represents characteristics of a health state of the pipeline;
in response to determining that the first validation value is not valid, causing to be displayed in a graphical user interface the first validation value; and
in response to determining that the second validation value is not valid, causing to be displayed in the graphical user interface the second validation value.

18. The one or more non-transitory computer-readable media of claim 17 , further comprising instructions, which, when executed by one or more processors, cause:
causing to be displayed in the graphical user interface a summary card for the one or more subsystems that summarizes the first validation value and the second validation value.

19. The one or more non-transitory computer-readable media of claim 18 , wherein the summary card for the one or more subsystems further comprises a plurality of numerical values, wherein each numerical value indicates a number of validation values associated with a particular status type for

Assignees

• Palantir Technologies Inc

Inventors

• Rickard Jesse
• Maag Peter
• Newman Jared
• Mecocci Giulio
• Subbanarasimhia Harish
• Dumitran Adrian Marius
• Skrodzki Andrzej
• Scheinerman Jonah
• Slonim Gregory
• Antihi Alexandru Viorel

Classifications

• G06T11/26

  Drawing of charts or graphs · CPC title

• G06F11/3055

  Monitoring arrangements for monitoring the status of the computing system or of the computing system component, e.g. monitoring if the computing system is on, off, available, not available (error or fault processing without redundancy G06F11/0703; error detection or correction by redundancy in data representation G06F11/08; error detection or correction of the data by redundancy in operations G06F11/14; error detection or correction by redundancy in hardware G06F11/16) · CPC title

• G06F11/3093

  Configuration details thereof, e.g. installation, enabling, spatial arrangement of the probes · CPC title

• G06F11/3024

  where the computing system component is a central processing unit [CPU] · CPC title

• G06F11/328 Primary

  Computer systems status display (G06F11/327 takes precedence) · CPC title