Etl diagnostics

Patent metadata

Field	Value
Publication number	US-2017068595-A1
Application number	US-201615158129-A
Country	US
Kind code	A1
Filing date	May 18, 2016
Priority date	Sep 4, 2015
Publication date	Mar 9, 2017
Grant date	—

Abstract

A cloud-based ETL system provides error detection, error correction and reporting of data integration flows hosted by cloud services. Categories of errors are identified using one or more checks at different points of a data integration flow and one or more actions selected based at least in part on the error category. A determination can be made whether the error category is fault tolerant and one or more actions can be selected based at least in part on the error fault tolerance to correct the error, restart a flow, or generate a notification assisting a user to correct the error.

First claim

What is claimed is:

1 . A method, comprising:
identifying, by a computer system, an error in a cloud-based data integration scenario having data flowing from a source to a target; determining, by the computer system, whether the error is correctable; performing, by the computer system, one or more actions that correct the error based at least in part on a category of the error when the error is correctable; determining, by the computer system, whether the error is fault tolerant based at least in part on the category of the error when the error is not correctable; processing, by the computer system, a first portion of the data in a first manner that rejects the data and a second portion of the data in a second manner that transforms the data when the error is fault tolerant; and generating, by the computer system, one or more notifications relevant to the error.

2 . The method of claim 1 , wherein identifying the error in the cloud-based data integration scenario comprises determining one or more errors with respect to the source.

3 . The method of claim 1 , wherein identifying the error in the cloud-based data integration scenario comprises determining one or more errors with respect to the data.

4 . The method of claim 1 , wherein identifying the error in the cloud-based data integration scenario comprises determining one or more errors with respect to a flow between the source and the target.

5 . The method of claim 1 , wherein determining whether the error is correctable comprises determining whether the data can be corrected.

6 . The method of claim 1 , wherein determining whether the error is correctable comprises determining whether a system can be restarted.

7 . The method of claim 1 , wherein performing the one or more actions that correct the error based at least in part on the category of the error when the error is correctable comprises correcting the data at the source.

8 . The method of claim 1 , wherein performing the one or more actions that correct the error based at least in part on the category of the error when the error is correctable comprises correcting the data at the target.

9 . The method of claim 1 , wherein performing the one or more actions that correct the error based at least in part on the category of the error when the error is correctable comprises correcting intermediate data.

10 . The method of claim 1 , wherein generating the one or more notifications relevant to the error comprises generating a notification that enables access to one or more cloud-based tools for correcting the error.

11 . A system, comprising:
a memory configured to store computer-executable instructions; and a processor configured to access the memory and execute the computer-executable instructions to at least:
identify an error in a data integration scenario having data flowing from a source to a target;
determine whether the error is correctable;
perform one or more actions that correct the error based at least in part on a category of the error when the error is correctable;
determine whether the error is fault tolerant based at least in part on the category of the error when the error is not correctable; and
process a first portion of the data in a first manner that rejects the data and a second portion of the data in a second manner that transforms the data when the error is fault tolerant.

12 . The system of claim 11 , wherein the error in the data integration scenario is identified based at least in part on one or more errors with respect to at least one of the source, the data, or a flow between the source and the target.

13 . The system of claim 11 , wherein the error is determined to be correctable based at least in part on determining whether at least one of the data can be corrected or a system can be restarted.

14 . The system of claim 11 , wherein the one or more actions that correct the error based at least in part on the category of the error when the error is correctable are performed by at least one of correcting the data at the source, correcting the data at the target, or correcting intermediate data.

15 . The system of claim 11 , wherein the computer-executable instructions are further executed to at least generate one or more notifications relevant to the error that enables access to one or more web service tools for correcting the error.

16 . A computer-readable medium configured to store computer-executable instruction that, when executed by one or more computer systems, cause the one or more computer systems to perform instructions comprising:
identifying an error in a data integration scenario having data flowing from a source to a target; determining whether the error is correctable; performing one or more actions that correct the error based at least in part on a category of the error when the error is correctable; determining whether the error is fault tolerant based at least in part on the category of the error when the error is not correctable; and processing a first portion of the data in a first manner that rejects the data and a second portion of the data in a second manner that transforms the data when the error is fault tolerant.

17 . The computer-readable medium of claim 16 , wherein the instructions further comprise generating one or more notifications relevant to the error.

18 . The computer-readable medium of claim 17 , wherein generating the one or more notifications relevant to the error comprises generating a notification that enables access to one or more cloud-based tools for correcting the error.

19 . The computer-readable medium of claim 16 , wherein identifying the error in the cloud-based data integration scenario comprises determining one or more errors with respect to a flow between the source and the target.

20 . The computer-readable medium of claim 16 , wherein performing the one or more actions that correct the error based at least in part on the category of the error when the error is correctable comprises correcting intermediate data.

Assignees

• Oracle Int Corp

Inventors

• Nautiyal Ispati
• Balu Rajesh

Classifications

• G06F2201/805

  Real-time · CPC title

• G06F11/1417

  Boot up procedures · CPC title

• G06F2201/85

  Active fault masking without idle spares · CPC title

• G06F11/1076 Primary

  Parity data used in redundant arrays of independent storages, e.g. in RAID systems · CPC title

• G06F16/254

  Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses · CPC title