Systems and methods for correlating errors to processing steps and data records to facilitate understanding of errors

Patent metadata

Field	Value
Publication number	US-10338986-B2
Application number	US-201615337014-A
Country	US
Kind code	B2
Filing date	Oct 28, 2016
Priority date	Oct 28, 2016
Publication date	Jul 2, 2019
Grant date	Jul 2, 2019

Abstract

A computer program file or a sequence of processing steps from an authoring environment of a data processing system is used to generate a graph including nodes representing the processing steps is generated. Unique identifiers (tags) are assigned to the nodes. One or more processing steps are optionally modified while preserving the original identifiers. If an error occurs during execution, error information is saved including the processing step that failed and data records that triggered the failure. The identifier of the failed processing step is attached to the error information. The error information is collected and processed. The collected error information is used to correlate the error to the failed processing step and to data records triggering the error using the identifier of the failed processing step recorded with the error information. The correlated error information indicates cause and location of the error.

First claim

What is claimed is:

1. A system comprising:
a processor and memory; and
machine readable instructions, when executed by the processor and memory, configured to:
receive one or more commands from a computer program file;
assign a first set of identifiers to different portions of the computer program file to link errors occurring during execution of one or more of a plurality of processing steps to corresponding portions of the computer program file;
execute the plurality of processing steps to process data in a data processing system distributed over a plurality of nodes in a cluster, wherein each of the nodes is a computing device;
compose a graph including vertices representing the plurality of processing steps;
assign a second set of identifiers to the vertices;
collect, from the plurality of nodes, in response to an error occurring on executing a first processing step of the processing steps on the plurality of nodes, information about the error stored on the plurality of nodes, the information about the error including a first identifier associated with one of the vertices representing the first processing step in the graph;
process the information about the error from the plurality of nodes to correlate the error to the first processing step based on the first identifier that is associated with the first processing step and that is included in the information about the error stored on the plurality of nodes;
generate, based on the processed information, correlation between the error and the first processing step, wherein the correlation between the error and the first processing step indicates a cause and a location of the error; and
in response to one or more of the plurality of processing steps being modified by reordering, consolidating, or discarding the one or more steps prior to executing the plurality of processing steps, retain identifiers of the one or more processing steps being modified with the modified one or more processing steps,
wherein the retained identifiers are used to correlate one or more errors to the one or more processing steps in the event of the modification.

2. The system of claim 1 wherein the machine readable instructions are further configured to:
collect, from the plurality of nodes in the cluster, in response to the error occurring on executing the first processing step on the plurality of nodes in the cluster, one or more records of data triggering the error included in the information about the error stored on the plurality of nodes in the cluster;
process the information about the error from the plurality of nodes in the cluster including the one or more records of data triggering the error to correlate the error to the one or more records of data triggering the error based on the first identifier that is associated with the first processing step and that is included in the information about the error; and
generate, based on the processed information, correlation between the error and the one or more records of data triggering the error to provide additional information about the cause and the location of the error.

3. The system of claim 1 wherein the machine readable instructions are further configured to assign the first set of identifiers to different portions of text in the computer program file to link the errors occurring during execution of the one or more of the plurality of processing steps to corresponding portions of the text in the computer program file.

4. The system of claim 1 wherein the machine readable instructions are further configured to execute the plurality of processing steps by compiling one or more commands for processing the data and to associate the plurality of processing steps and the corresponding identifiers to respective portions of the one or more commands to identify a portion of the one or more commands causing the error.

5. The system of claim 1 wherein the machine readable instructions are further configured to:
generate information at each of the plurality of nodes for less than or equal to a predetermined number of errors;
collect information from each of the plurality of nodes about less than or equal to a predetermined number of tags; or
provide the correlation for less than or equal to a predetermined number of errors.

6. The system of claim 1 wherein the plurality of processing steps process the data in a batch processing mode.

7. The system of claim 1 wherein the plurality of processing steps process the data in a streaming mode.

8. A method for correlating errors in a data processing system, the method comprising:
receiving one or more commands from a computer program file;
assigning a first set of identifiers to different portions of the computer program file to link errors occurring during execution of one or more of a plurality of processing steps to corresponding portions of the computer program file;
executing the plurality of processing steps to process data in a data processing system distributed over a plurality of nodes in a cluster, each of the nodes being a computing device;
composing a graph including vertices representing the plurality of processing steps;
assigning a second set of identifiers to the vertices;
collecting, from the plurality of nodes, in response to an error occurring on executing a first processing step of the processing steps on the plurality of nodes, information about the error stored on the plurality of nodes, the information about the error including a first identifier associated with one of the vertices representing the first processing step in the graph;
processing the information about the error from the plurality of nodes to correlate the error to the first processing step based on the first identifier that is associated with the first processing step and that is included in the information about the error stored on the plurality of nodes;
generating, based on the processed information, correlation between the error and the first processing step, wherein the correlation between the error and the first processing step indicates a cause and a location of the error;
in response to modifying one or more of the plurality of processing steps by reordering, consolidating, or discarding the one or more steps prior to executing the plurality of processing steps, retaining identifiers of the one or more processing steps being modified with the modified one or more processing steps; and
using the retained identifiers to correlate one or more errors to the one or more processing steps in the event of the modification.

9. The method of claim 8 further comprising:
collecting, from the plurality of nodes in the cluster, in response to the error occurring on executing the first processing step on the plurality of nodes in the cluster, one or more records of data triggering the error included in the information about the error stored on the plurality of nodes in the cluster;
processing the information about the error from the plurality of nodes in the cluster including the one or more records of data triggering the error to correlate the error to the one or more records of data triggering the error based on the first identifier that is associated with the first processing step and that is included in the information about the error; and
generating, based on the processed information, correlation between the error and the one or more records of data triggering the error to provide additional information about the cause and the location of the error.

10. The method of claim 8 further comprising:
assigning the first set of identifiers to different portions of the text in the computer program file to link errors occurring during execution of the one or more of the plurali

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Grealish Kevin David James
• Garden Euan
• Lee Craig Allen
• Gomez Jose Ramon Calzada
• Coulson Michael James

Classifications

• G06F11/0775 Primary

  Content or structure details of the error report, e.g. specific table structure, specific error fields · CPC title

• G06F11/0784

  Routing of error reports, e.g. with a specific transmission path or data flow · CPC title

• G06F11/3628

  of optimised code (optimisation G06F8/443) · CPC title

• G06F11/0724

  in a multiprocessor or a multi-core unit (multiprocessors per se G06F15/80) · CPC title

• G06F11/0709

  in a distributed system consisting of a plurality of standalone computer nodes, e.g. clusters, client-server systems · CPC title