In-memory databases for integration testing of extraction, transformation, and/or loading jobs

Patent metadata

Field	Value
Publication number	US-12067026-B2
Application number	US-202217891554-A
Country	US
Kind code	B2
Filing date	Aug 19, 2022
Priority date	Aug 19, 2022
Publication date	Aug 20, 2024
Grant date	Aug 20, 2024

Abstract

A method, apparatus, and computer-readable medium are described that provide integrated testing and validation of multi-phase instructions before deployment. Aspects of the disclosure relate to testing ETL instructions using an efficient process that individually and/or holistically validates the ETL instructions and, using a random number generator, varies various aspects of the source datasets. A benefit of running the combination of instructions includes finding errors that are not apparent during testing of each instruction phase separately. Instructions may be separately provided that identify the framework of an in-memory source dataset (e.g., a quantity of rows, a quantity of columns, types of the various columns, data for each record, and the like). The instructions for the framework may also identify a variability of one or more of the items of the framework.

First claim

What is claimed is:

1. A computer-implemented method for testing instructions the method comprising:
receiving, from a storage, first database instructions comprising:
first extraction instructions configured to extract source content,
first transform instructions configured to transform the extracted source content, and
first load instructions configured to load the transformed source content as target content;

generating, in a memory of a computing system and from sampling, via a random number generator, a source database description and a target database description, a source database framework and a target database framework;
receiving first test information, wherein the first test information describes source database content and variability of the source database content;
generating, based on:
the source database framework,
the first test information, and
the random number generator,

a source database, wherein the source database comprises first records and first fields, and wherein the generating the source database comprises sampling, via the random number generator, the first test information to generate the first records;
generating, based on:
the source database,
the first database instructions, and
the target database framework,

a target database comprising target records and target fields;
validating at least a portion of the target database against a corresponding portion of the source database; and
generating, based on:
a successful validation of the at least the portion of the target database against the corresponding portion of the source database,
the variability of the source database content, and
the random number generator,
a modified source database,

wherein the modified source database comprises second records and second fields,
wherein the generating the modified source database comprises sampling, via the random number generator, the first test information to generate the second records, and
wherein the source database and the modified source database differ in at least one of records or fields.

2. The computer-implemented method of claim 1 , wherein the source database differs from the modified source database in quantities of records.

3. The computer-implemented method of claim 1 , wherein the source database differs from the modified source database in values in respective records.

4. The computer-implemented method of claim 1 , wherein the source database differs from the modified source database in quantities of fields.

5. The computer-implemented method of claim 1 , wherein the source database differs from the modified source database in types of fields.

6. The computer-implemented method of claim 1 , wherein the validating comprises:
comparing a quantity of the first records of the source database with a quantity of target records of the target database,
wherein generating the modified source database is further based on the comparison of the quantity of the first records of the source database with a quantity of target records of the target database.

7. The computer-implemented method of claim 1 , wherein the validating comprises:
comparing values between:
a selected field of a record of the first records of the source database with
a corresponding field of a corresponding record of the target records of the target database,

wherein generating the modified source database is further based on the comparison of the selected field and the corresponding field.

8. The computer-implemented method of claim 1 , further comprising:
validating at least a second portion of the target database against a corresponding second portion of the source database;
generating, based on a failed validation of the at least the second portion of the target database against the corresponding second portion of the source database, an indication of the failed validation; and
receiving, based on the generation of the indication of the failed validation, modified first extraction instructions.

9. The computer-implemented method of claim 8 , further comprising:
outputting the indication of the failed validation with the at least the second portion of the target database and the corresponding second portion of the source database.

10. The computer-implemented method of claim 8 , further comprising:
generating, based on:
the modified source database,
the first database instructions, and
target database framework,
a modified target database comprising modified target records and modified target fields;

validating at least a portion of the modified target database against a corresponding portion of the modified source database; and
outputting a result of a validation of the at least the portion of the modified target database against the corresponding portion of the modified source database.

11. An apparatus comprising:
one or more processors; and
memory storing instructions that, when executed by the one or more processors, cause the apparatus to:
receive, from a storage, first database instructions comprising:
first extraction instructions configured to extract source content,
first transform instructions configured to transform the extracted source content, and
first load instructions configured to load the transformed source content as target content;

generate, in a memory of a computing system and from sampling, via a random number generator, a source database description and a target database description, a source database framework and a target database framework;
receive first test information, wherein the first test information describes source database content and variability of the source database content:
generate, based on:
the source database framework,
the first test information, and
the random number generator,

a source database, wherein the source database comprises first records and first fields, and wherein the instructions to generate of the source database cause the apparatus to sample the first test information to generate the first records;
generate, based on:
the source database,
the first database instructions, and
the target database framework,

a target database comprising target records and target fields;
validate at least a portion of the target database against a corresponding portion of the source database; and
generate, based on:
a successful validation of the at least the portion of the target database against the corresponding portion of the source database,
the variability of the source database content, and
the random number generator,
a modified source database,

wherein the modified source database comprises second records and second fields, and
wherein the instructions to the generate the modified source database cause the apparatus to sample the first test information to generate the second records.

12. The apparatus of claim 11 , wherein the source database differs from the modified source database in quantities of records.

13. The apparatus of claim 11 , wherein the source database differs from the modified source database in values in respective records.

14. The apparatus of claim 11 , wherein the source database differs from the modified source database in quantities of fields.

15. The apparatus of claim 11 , wherein the source database differs from the modified source database in types of fields.

16. The apparatus of claim 11 , wherein the instructions to validate further cause the apparatus to:
compare a quantity of the first records of the source database with a quantity of target records of the target database,
wherein the genera

Assignees

• Capital One Services Llc

Inventors

• Konduru Anil
• Deshpande Amit Ramesh
• Forsberg Dillon
• Sadananda Anvitha Banakal
• Mony Jameskutty

Classifications

• G06F11/3684

  for test design, e.g. generating new test cases · CPC title

• G06F16/27

  Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor · CPC title

• G06F16/254 Primary

  Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses · CPC title