System for augmenting and joining multi-cadence datasets

Patent metadata

Field	Value
Publication number	US-11714812-B2
Application number	US-202117315624-A
Country	US
Kind code	B2
Filing date	May 10, 2021
Priority date	May 10, 2021
Publication date	Aug 1, 2023
Grant date	Aug 1, 2023

Abstract

A computing system may comprise a server system, a database, and one or more data sources having different cadences, such as a batch data source and a real-time data source. The server system may generate a first dataset based on data from the batch data source, and may generate a second dataset based on data received from the real-time data source. The server system may determine metadata associated with the real-time data source. Based on the metadata, the server system may generate a database table representation of the real-time data source. The server system may be configured to perform a relational join on the first and second datasets. Such a relational join may define a namespace that is based on the first and second datasets.

First claim

What is claimed is:

1. A computer-implemented method, comprising:
receiving streaming data from a real-time data source and batch data from a batch data source, wherein the batch data is structured as batch data rows and batch data columns;
converting the streaming data into a format comprising streaming data rows and streaming data columns;
storing, in a database, the batch data and the converted streaming data;
generating, based on the stored batch data, a first user interface and a second user interface, wherein the first user interface comprises a first selectable list of the batch data columns and a first region for designating a first key column of the batch data columns, and wherein the second user interface comprises a second selectable list of the converted streaming data columns and a second region for designating a second key column of the converted streaming data columns;
receiving, based on a first user interaction with the first user interface, a selected first key column of the batch data columns and a first namespace selection, wherein the first namespace selection comprises a first selection of less than all of the batch data columns;
receiving, based on a second user interaction with the second user interface, a selected second key column of the streaming data columns and a second namespace selection, wherein the second namespace selection comprises a second selection of less than all of the streaming data columns;
generating, based on the first namespace selection and the second namespace selection, a third user interface comprising a combined namespace of the first namespace selection and the second namespace selection;
performing, based on a third user interaction with the third user interface, a relational join, based on the selected first key column and the selected second key column, of the batch data rows in the first namespace selection of the batch data columns against the converted streaming data rows in the second namespace selection of the converted streaming data columns; and
outputting, based on performing the relational join, results of the relational join.

2. The computer-implemented method of claim 1 , further comprising:
receiving on-demand data from an on-demand data source; and
modifying the converted streaming data based on the on-demand data, wherein storing the converted streaming data in the database comprises storing the modified converted streaming data in the database.

3. The computer-implemented method of claim 2 , wherein the on-demand data comprises application programming interface (API) data from an API data source.

4. The computer-implemented method of claim 1 , further comprising:
determining metadata for the real-time data source, wherein converting the streaming data into the format comprising streaming data rows and streaming data columns is based on the metadata.

5. The computer-implemented method of claim 1 , wherein:
receiving the batch data from the batch data source is based on a user input selection of the batch data source, and
receiving the streaming data from the real-time data source is based on a user input selection of the real-time data source.

6. The computer-implemented method of claim 1 , further comprising:
determining a cadence of the real-time data source, wherein the cadence indicates a frequency at which the real-time data source is updated; and
based on the cadence:
receiving additional streaming data from the real-time data source; and
updating the combined namespace based on the received additional streaming data.

7. The computer-implemented method of claim 1 , further comprising:
causing output, for display, of preview data from the real-time data source,
wherein the preview data comprises one or more of:
at least one column name of the batch data columns; or
at least one attribute of at least one column of the real-time data source.

8. A computing device comprising:
one or more processors; and
memory storing instructions that, when executed by the one or more processors, cause the computing device to:
receive indications of a plurality of real-time data sources;
select, based on a first user input, first real-time data of a first real-time data source of the plurality of real-time data sources;
select, based on a second user input, second real-time data from a second real-time data source of the plurality of real-time data sources;
convert the first real-time data into a format comprising first streaming data rows and first streaming data columns;
store the converted first real-time data in a first database table;
convert the second real-time data into a format comprising second streaming data rows and second streaming data columns;
store the second converted real-time data in a second database table;
generate, based on the first database table, a first user interface and, based on the second database table, a second user interface, wherein the first user interface comprises a first selectable list of the first streaming data columns and a first region for designating a first key column of the first streaming data columns, and wherein the second user interface comprises a second selectable list of the second streaming data columns and a second region for designating a second key column of the second streaming data columns;
receive, based on a first user interaction with the first user interface, a selected first key column and a first namespace selection, wherein the first namespace selection comprises a first selection of less than all of the first streaming data columns;
receive, based on a second user interaction with the second user interface, a selected second key column and a second namespace selection, wherein the second namespace selection comprises a second selection of less than all of the second streaming data columns;
generate, based on the first namespace selection and the second namespace selection, a third user interface comprising a combined namespace of the first namespace selection and the second namespace selection;
perform, based on a third user interaction with the third user interface, a relational join, based on the selected first key column and the selected second key column, of the first streaming data rows in the first namespace selection of the first streaming data columns against the second streaming data rows in the second namespace selection of the second streaming data columns; and
output, based on relational join, results of the relational join.

9. The computing device of claim 8 , wherein the instructions, when executed by the one or more processors, further cause the computing device to:
receive on-demand data from an on-demand data source; and
modify the converted first real-time data based on the on-demand data, wherein storing the converted first real-time data comprises storing the modified first real-time data in the first database table.

10. The computing device of claim 8 , wherein the instructions, when executed by the one or more processors, further cause the computing device to:
output, for display, preview data from the first real-time data source,
wherein the preview data comprises at least one column name that corresponds to at least one column of the first real-time data source.

11. The computing device of claim 8 , wherein the instructions, when executed by the one or more processors, further cause the computing device to:
determine first metadata for the first real-time data source and second metadata for the second real-time data source, wherein:
converting the first real-time data is based on the first metadata; and
converting the second real-time data is based on the second metadata.

Assignees

• Capital One Services Llc

Inventors

• Kancharla Gopi
• Tandra Praveen
• Kumar Parvesh
• Sammeta Suresh
• Momin Zainil
• Mupparapu Srinivas
• Doubet Dustin

Classifications

• G06F16/2456 Primary

  Join operations · CPC title

• G06F16/2379

  Updates performed during online database operations; commit processing · CPC title