Data model driven design of data pipelines configured on a cloud platform

Patent metadata

Field	Value
Publication number	US-11343142-B1
Application number	US-202117402379-A
Country	US
Kind code	B1
Filing date	Aug 13, 2021
Priority date	Apr 15, 2021
Publication date	May 24, 2022
Grant date	May 24, 2022

Abstract

A system performs continuous delivery of a data pipeline on a cloud platform. The system receives a specification of the data pipeline comprising data pipeline units. The system generates a deployment package for each data pipeline unit for a cloud platform. The system provisions computing infrastructure on the cloud platform according to the system configuration of the data pipeline unit. The data pipeline may be implemented as a data mesh. The data pipeline generates one or more data models. The system receives a schema representing a modification in a data model based on a change in the requirements of a consumer system. The system determines the changes to the data pipeline based on the received schema and reconfigures the data pipeline to generate the modified data model. The system manages access control of data to minimize the exposure to data in case of accidental or malicious data breach.

First claim

What is claimed is:

1. A computer-implemented method for modifying data pipelines configured on a cloud platform, the method comprising:
receiving a specification of a data pipeline comprising a plurality of data pipeline units, the data pipeline generating output representing a data model associated with an application, the data model conforming to an original interoperability schema, wherein an interoperability schema defines format for exchanging data across computer systems independent of the format of storage of the data in the computer systems;
for each data pipeline unit of the data pipeline, generating instructions for configuring the data pipeline unit on the cloud platform from the specification of the data pipeline unit;
deploying the data pipeline on the cloud platform using the generated instructions;
receiving a differential interoperability schema representing a modification to the data model associated with the application, the differential interoperability schema specifying differences compared to the original interoperability schema;
generating a set of instructions to modify the data pipeline such that the modified data pipeline generates outputs representing a modified data model conforming to a composition of the original interoperability schema and the differential interoperability schema;
reconfiguring the data pipeline on the cloud platform based on the generated instructions; and
executing the reconfigured data pipeline on the cloud platform to provide data conforming to the modified data model to the application.

2. The computer-implemented method of claim 1 , wherein generating the set of instructions for modifying the data pipeline comprises:
identifying a data pipeline unit of the data pipeline that generates all elements of the modified interoperability schema, wherein the data pipeline unit generates a first set of outputs; and
modifying the identified data pipeline unit to generate a second set of outputs representing the modified data model.

3. The computer-implemented method of claim 1 , wherein generating the set of instructions for modifying the data pipeline comprises:
identifying a first data pipeline unit of the data pipeline that generates a first subset of elements of the modified interoperability schema and identifying a second data pipeline unit of the data pipeline that generates a second subset of elements of the modified interoperability schema; and
generating a third data pipeline unit that receives inputs from the first data pipeline unit and the second data pipeline unit and generates a set of outputs representing the modified data model.

4. The computer-implemented method of claim 1 , wherein the application is associated with events, executing the reconfigured data pipeline on the cloud platform to provide data conforming to the modified data model to the application comprises:
receiving indication of an occurrence of an event associated with the application;
receiving a filter associated with the event;
applying the filter to identify one or more records;
triggering execution of the data pipeline for the identified one or more records to generate outputs conforming to the modified data model; and
providing the generated output to the application.

5. The computer-implemented method of claim 4 , further comprising:
saving the generated outputs in a storage unit of one or more data pipeline units for batch processing;
receiving a request to generate a report describing events that occurred during a time interval;
generating a report, wherein the generated report describes a set of events including the event.

6. The computer-implemented method of claim 1 , wherein generated instructions for the data pipeline comprise instructions for each data pipeline unit, wherein the instructions for a data pipeline unit comprise:
a system configuration for the data pipeline unit, the system configuration comprising instructions for configuring: one or more storage units on the cloud platform, a cluster of servers for execution of the data pipeline unit on the cloud platform, and one or more processing engines for executing instructions of the data pipeline unit, and
a deployment package comprising: data flow instructions for orchestrating the flow of data across resources of the data pipeline unit, and a transformation processing instructions package for performing the one or more data transformations of the data pipeline unit.

7. The computer-implemented method of claim 1 , wherein an output of the data pipeline is one of:
a data stream that provides data elements at various time intervals; or
a batch input that provides a data set comprising a plurality of data elements at one point in time.

8. The computer-implemented method of claim 1 , wherein the differential interoperability schema is a first differential interoperability schema, the modification to the data model is a first modification to the data model, the set of instructions is a first set of instructions, and the modified data model is a first modified data model:
receiving a second differential interoperability schema representing a second modification to the data model associated with the application, the second differential interoperability schema specifying differences compared to the original interoperability schema;
generating a second set of instructions to modify the data pipeline such that the modified data pipeline generates outputs representing a second modified data model conforming to a composition of the original interoperability schema and the second differential interoperability schema;
reconfiguring the data pipeline on the cloud platform based on the second set of generated instructions; and
executing the reconfigured data pipeline on the cloud platform to provide data conforming to both the first modified data model and the second modified data model.

9. The computer-implemented method of claim 1 , wherein the differential interoperability schema specifies a uniform resource locator (URL) identifying a set of values for an element, wherein generating the set of instructions comprises:
generating a transformation for a data pipeline unit, wherein the transformation excludes records that have elements outside the set of values.

10. The computer-implemented method of claim 1 , wherein the differential interoperability schema specifies a uniform resource locator (URL) specifying a procedure associated with an element specified in the original interoperability schema, wherein generating the set of instructions comprises:
generating a transformation for a data pipeline unit, wherein the transformation applies the procedure to data of the element.

11. The computer-implemented method of claim 1 , wherein the original interoperability schema specifies a first data type for an element, and the differential schema specifies a second data type for the element, wherein generating the set of instructions comprises:
generating a transformation for a data pipeline unit, wherein the transformation converts data of the element from the first data type to the second data type.

12. The computer-implemented method of claim 1 , wherein the differential interoperability schema specifies a specifying a regular expression representing a constraint for an element, wherein generating the set of instructions comprises:
generating a transformation for a data pipeline unit, wherein the transformation excludes records that fail to match the regular expression.

13. The computer-implemented method of claim 1 , wherein the differential interoperability schema specifies a maximum size for an element to be zero, wherein generating the

Assignees

• Humana Inc

Inventors

• Wang Huaxia
• Sriram Padma
• Yao Yuan
• Zhang Bing

Classifications

• G06F8/60

  Software deployment · CPC title

• G06N7/00

  Computing arrangements based on specific mathematical models · CPC title

• G06F16/283

  Multi-dimensional databases or data warehouses, e.g. MOLAP or ROLAP · CPC title

• H04L41/0813 Primary

  characterised by the conditions triggering a change of settings · CPC title

• H04L63/10

  for controlling access to devices or network resources · CPC title