Systems and methods for data storage and processing

Patent metadata

Field	Value
Publication number	US-11397744-B2
Application number	US-201916517253-A
Country	US
Kind code	B2
Filing date	Jul 19, 2019
Priority date	Jul 19, 2018
Publication date	Jul 26, 2022
Grant date	Jul 26, 2022

Abstract

Systems and methods for processing data are provided. The system may include at least a processor and a non-transient data memory storage, the data memory storage containing machine-readable instructions for execution by the processor, the machine-readable instructions configured to, when executed by the processor, provide an information delivery platform configured to: extract raw data from a plurality of source systems; load and store the raw data at a non-transient data store; receive a request to generate data for consumption for a specific purpose; in response to the request, select a set of data from the raw data based on a data map; transform the selected set of data into a curated set of data based on the data map; and transmit the curated set of data to a channel for consumption.

First claim

What is claimed is:

1. A system for processing data, comprising at least a processor and a non-transient data memory storage, the data memory storage containing machine-readable instructions for execution by the processor, the machine-readable instructions configured to, when executed by the processor, provide an information delivery platform configured to:
receive raw data from a plurality of source systems;
load and store the raw data at one or more appliances, the one or more appliances providing a non-transient data store and computation engine;
receive a request to generate data for consumption, the request indicating a specific purpose for the consumption;
retrieve a data map corresponding to at least one attribute of the request, the data map indicating the raw data to be selected that correspond to the at least one attribute of the request select a set of data from the raw data based on the request in accordance with the data map;
transform, using computation engine at the one or more appliances, the set of data into a transformed dataset for consumption, the set of data being maintained at the non-transient data store during transformation, wherein the data map indicates a common data field to be created within the curated set of data that includes at least a portion of the selected raw data and further indicates code to transform the selected data into the curated set of data; and
store the transformed dataset at the non-transient data store.

2. The system of claim 1 , wherein the non-transient data store is distributed across a network of appliances.

3. The system of claim 1 , wherein the selection of the set of data comprises aggregating a portion of the raw data using the data map.

4. The system of claim 1 , wherein the raw data is received sequentially from the plurality of source systems.

5. The system claim 1 , wherein the raw data is loaded and stored sequentially according to the one or more plurality of source systems the raw data was received from, a sequential order based on timing data from the source systems relating to the availability of the raw data.

6. The system of claim 1 , wherein the information delivery platform is further configured to generate one or more data models of the raw data, selected set of data, or transformed dataset, the one or more data models defining attributes descriptive of data fields to describe features or aspects of the raw data, selected set of data, or transformed dataset.

7. The system of claim 6 , wherein the one or more data models encode data for using the raw data, selected set of data, or transformed dataset.

8. The system of claim 7 , wherein the one or more data models is generated based on machine learning rules.

9. The system of claim 8 , wherein the data map is populated based on one or more data models.

10. A system for processing data, comprising at least a processor and a non-transient data memory storage, the data memory storage containing machine-readable instructions for execution by the processor, the machine-readable instructions configured to, when executed by the processor, provide an information delivery platform configured to:
extract raw data from a plurality of source systems;
load and store the raw data at a non-transient data store;
receive a request to generate data for consumption, the request indicating a specific purpose for the consumption;
in response to the request, retrieve a data map corresponding to at least one attribute of the request, the data map indicating a list of data columns within the raw data to be selected that corresponds to the at least one attribute of the request;
select a set of data from the raw data based on the data map;
transform the selected set of data into a curated set of data based on the data map, wherein the data map indicates a common data field to be created within the curated set of data that includes at least a portion of the selected raw data and further indicates code to transform the selected data into the curated set of data; and
transmit the curated set of data to a channel for the consumption.

11. The system of claim 10 , wherein the raw data are stored at the non-transient data store in a data format that is identical to a source data format of the raw data in the plurality of source systems.

12. The system of claim 10 , wherein the data map further comprises a visual graph linking one or more data columns of the raw data to one or more data fields of the curated set of data.

13. The system of claim 1 , wherein the one or more appliances are integrated into the information delivery platform and configured to access data in the non-transient data store.

14. The system of claim 1 , wherein the raw data from the plurality of sources is in a corresponding plurality of source data formats, wherein the transformed data is in a common data format based on the request.

15. The system of claim 1 , wherein the processor generates an action based on real-time transaction data and the transformed data set.

16. The system of claim 10 , wherein the data map is generated based on data attributes stored in a metadata database.

17. The system of claim 15 , wherein the data map is generated through machine learning techniques.

18. The system of claim 10 , wherein the specific purpose relates to generating visual elements for an interface to display information to a specific group of users of the information delivery platform.

19. A computer-implemented method for executing by a processor, the method comprising:
extracting, by the processor, raw data from a plurality of source systems;
loading and storing, by the processor, the raw data at a non-transient data store;
receiving, by the processor, a request to generate data for consumption for a specific purpose;
in response to the request, retrieving, by the processor, a data map corresponding to at least one attribute of the request, the data map indicating a list of data columns within the raw data to be selected that corresponds to the at least one attribute of the request;
selecting, by the processor, a set of data from the raw data based on the data map;
transforming the selected set of data into a curated set of data based on the data map, wherein the data map indicates a common data field to be created within the curated set of data that includes at least a portion of the selected raw data and further indicates code to transform the selected data into the curated set of data; and
transmitting, by the processor, the curated set of data to a channel for consumption.

20. The method of claim 19 , wherein the specific purpose comprises displaying information to a specific group of users of the information delivery platform.

21. The method of claim 19 , wherein the raw data are stored at the non-transient data store in a data format that is identical to a source data format of the raw data in the plurality of source systems.

22. The method of claim 19 , wherein the data map further comprises a visual graph linking one or more data columns of the raw data to one or more data fields of the curated set of data.

23. The method of claim 22 , wherein the data map is generated based on data attributes stored in a metadata database.

24. The method of claim 22 , comprising generating the data map through machine learning techniques.

Assignees

• Bank Of Montreal

Inventors

• Wan Ching Leong
• Wang Jun

Classifications

• G06F16/254 Primary

  Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses · CPC title

• G06N5/022

  Knowledge engineering; Knowledge acquisition · CPC title

• G06N20/00

  Machine learning · CPC title