Data pipeline controller

Patent metadata

Field	Value
Publication number	US-11983189-B2
Application number	US-202016831999-A
Country	US
Kind code	B2
Filing date	Mar 27, 2020
Priority date	Mar 27, 2020
Publication date	May 14, 2024
Grant date	May 14, 2024

Abstract

A processing system including at least one processor may obtain a first ontology of a first type of data pipeline component, map the first ontology to a second ontology for a second type of data pipeline component that is stored in a catalog of data pipeline component types, provide a second data schema for the second type of data pipeline component as a template for a first data schema for the first type of data pipeline component, and add the first type of data pipeline component to the catalog of data pipeline component types, where the adding comprises storing the first ontology and the first data schema for the first type of data pipeline component in the catalog of data pipeline component types.

First claim

What is claimed is:

1. A method comprising:
obtaining, by a processing system including at least one processor deployed in a communication network, a first ontology of a first type of data pipeline component;
mapping, by the processing system, the first ontology to a second ontology for a second type of data pipeline component that is stored in a catalog of data pipeline component types;
providing, by the processing system, a second data schema for the second type of data pipeline component as a template for a first data schema for the first type of data pipeline component;
adding, by the processing system, the first type of data pipeline component to the catalog of data pipeline component types, wherein the adding comprises storing the first ontology and the first data schema for the first type of data pipeline component in the catalog of data pipeline component types;
obtaining, by the processing system, a request for a delivery of a data set to at least one destination;
mapping, by the processing system, the request to a first information model from among a plurality of information models, wherein the first information model comprises a flow sequence for a data pipeline;
selecting, by the processing system, a plurality of data schemas of a plurality of data pipeline component types from the catalog of data pipeline component types in accordance with the first information model;
determining, by the processing system, a plurality of data pipeline components, wherein the plurality of data pipeline components comprises a plurality of host devices deployed in the communication network; and
configuring, by the processing system, the plurality of data pipeline components in accordance with the plurality of data schemas into the data pipeline via one or more commands to one or more respective data pipeline components of the plurality of data pipeline components as defined in one or more respective data schemas of the plurality of data schemas associated with the one or more respective data pipeline components, wherein the data pipeline is to deliver the data set to the at least one destination.

2. The method of claim 1 , further comprising:
presenting the template to an operator.

3. The method of claim 2 , further comprising:
obtaining an approval of the operator to deploy the template as the first data schema.

4. The method of claim 2 , further comprising:
obtaining at least one change to the template from the operator; and
changing the template in accordance with the at least one change, wherein the first data schema that is stored in the catalog of data pipeline component types comprises the template that is changed.

5. The method of claim 1 , wherein the mapping comprises:
determining a similarity between the second type of data pipeline component and the first type of data pipeline component, wherein the similarity is based upon a congruence between the first ontology of the first type of data pipeline component and the second ontology of the second type of data pipeline component.

6. The method of claim 5 , wherein the congruence is based upon a number of matches between at least one of: classes, properties, or class restrictions of the first ontology and at least one of: classes, properties, or class restrictions of the second ontology.

7. The method of claim 6 , wherein the mapping comprises:
determining that a measure of the congruence between the first ontology and the second ontology exceeds measures of congruencies between the first ontology and a plurality of other ontologies of a plurality of other types of data pipeline components stored in the catalog of data pipeline component types.

8. The method of claim 1 , further comprising:
identifying that the first information model is impacted by the adding of the first type of data pipeline component to the catalog of data pipeline component types; and
providing at least one suggestion to an operator comprising at least one of: a suggestion to modify the first information model to incorporate the first type of data pipeline component or a suggestion to create a new information model based upon the first information model and that incorporates the first type of data pipeline component.

9. The method of claim 8 , wherein the first information model further comprises data attribute relationships.

10. The method of claim 8 , wherein the identifying comprises:
determining that the first information model includes at least one hook that identifies the second type of data pipeline component.

11. The method of claim 1 , wherein the first information model comprises first metadata relating to at least one of:
a name;
a region;
a task type; or
a technology.

12. The method of claim 11 , wherein the request comprises second metadata relating to at least one of:
the name;
the region;
the task type; or
the technology.

13. The method of claim 12 , wherein the mapping the request comprises:
mapping the request to the first information model based upon a congruence between the first metadata and the second metadata.

14. The method of claim 1 , wherein the first information model comprises hooks to the plurality of data schemas.

15. The method of claim 1 , further comprising:
determining whether an existing data pipeline is available to handle the request.

16. The method of claim 15 , wherein the determining and the configuring are performed when it is determined that no existing data pipeline is available to fulfill the request.

17. A non-transitory computer-readable medium storing instructions which, when executed by a processing system including at least one processor when deployed in a communication network, cause the processing system to perform operations, the operations comprising:
obtaining a first ontology of a first type of data pipeline component;
mapping the first ontology to a second ontology for a second type of data pipeline component that is stored in a catalog of data pipeline component types;
providing a second data schema for the second type of data pipeline component as a template for a first data schema for the first type of data pipeline component;
adding the first type of data pipeline component to the catalog of data pipeline component types, wherein the adding comprises storing the first ontology and the first data schema for the first type of data pipeline component in the catalog of data pipeline component types;
obtaining a request for a delivery of a data set to at least one destination;
mapping the request to a first information model from among a plurality of information models, wherein the first information model comprises a flow sequence for a data pipeline;
selecting a plurality of data schemas of a plurality of data pipeline component types from the catalog of data pipeline component types in accordance with the first information model;
determining a plurality of data pipeline components, wherein the plurality of data pipeline components comprises a plurality of host devices deployed in the communication network; and
configuring the plurality of data pipeline components in accordance with the plurality of data schemas into the data pipeline via one or more commands to one or more respective data pipeline components of the plurality of data pipeline components as defined in one or more respective data schemas of the plurality of data schemas associated with the one or more respective data pipeline components, wherein the data pipeline is to deliver the data set to the at least one destination.

18. An apparatus comprising:
a proce

Assignees

• At & T Ip I Lp

Inventors

• Fan James
• Aaron Jeffrey
• Agraharam Sanjay
• Gupta Arun
• Martens Michelle
• Polston Steven

Classifications

• G06F9/3005

  to perform operations for flow control · CPC title

• G06F16/212

  with details for data modelling support · CPC title

• G06F16/254 Primary

  Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses · CPC title

• G06F9/30079

  Pipeline control instructions, e.g. multicycle NOP · CPC title

• G06F9/3812

  with instruction modification, e.g. store into instruction stream · CPC title