Pipeline systems and methods for use in data analytics platforms

Patent metadata

Field	Value
Publication number	US-2022012241-A1
Application number	US-202017011731-A
Country	US
Kind code	A1
Filing date	Sep 3, 2020
Priority date	Jul 9, 2020
Publication date	Jan 13, 2022
Grant date	—

Abstract

A data analytics system including an append-only first data store accessible to multiple clients and a second data store is disclosed. The data analytics system can be configurable to, in response to receiving first instructions from a first target system of a first client, the first target system separate from the data analytics system, create a first pipeline between the append-only first data store and the second data store. The first pipeline can be configured according to the first instructions to generate a client-specific data object and store the client-specific data object in the second data store. The data analytics system can be configurable to tear down the first pipeline upon completion of storing the client-specific data object in the second data store.

First claim

What is claimed is:

1 . A data analytics system, comprising
an append-only first data store accessible to multiple clients; a second data store; and wherein the data analytics system is configurable to:
in response to receiving first instructions from a first target system of a first client, the first target system separate from the data analytics system:
create a first pipeline between the append-only first data store and the second data store, the first pipeline configured according to the first instructions to generate a client-specific data object and store the client-specific data object in the second data store;
teardown the first pipeline upon completion of storing the client-specific data object in the second data store; and

in response to receiving second instructions from a second target system of the first client, the second target system separate from the data analytics system:
create a second pipeline between the second data store and the second target system, the second pipeline configured according to the second instructions to generate query results using, at least in part, the client-specific data object and provide the query results to the second target system; and
teardown the second pipeline upon completing provision of the query results to the second target system.

2 . The data analytics system of claim 1 , wherein the first pipeline is configured according to the first instructions to generate the client-specific data object based on data from multiple distinct data sources in the append-only first data store.

3 . The data analytics system of claim 1 , wherein the first instructions comprise a declarative specification of the first pipeline.

4 . The data analytics system of claim 1 , wherein the first instructions comprise a template personalized with metadata.

5 . The data analytics system of claim 1 , wherein the first instructions comprise a function call and the data analytics system is configurable to generate first infrastructure instructions for creating the first pipeline to implement the function call.

6 . The data analytics system of claim 1 , wherein the first pipeline is configured to generate the client-specific data object by performing at least one of validation, aggregation, filtering, classification, transformation, or coding operations.

7 . The data analytics system of claim 1 , wherein the first pipeline is configured to associate a context with the client-specific data object.

8 . The data analytics system of claim 7 , wherein the context specifies at least one of the first client, information describing the first pipeline, or security information for the client-specific data object.

9 . The data analytics system of claim 1 , wherein the second pipeline is configured according to the second instructions to provide virtualized access to an external data source and generate the query results using, at least in part, the virtualized access to the external data source.

10 . The data analytics system of claim 1 , wherein the data analytics system is configurable to generate, in response to a privileged query, at least one of information describing data objects for multiple clients stored in the second data store, information describing first pipelines created by clients between the first and second data stores, or information describing second pipelines created by clients between the second data store and target systems of the clients.

11 . The data analytics system of claim 1 , wherein the data analytics system is configurable to load information received from external data sources into the first data store in response to privileged instructions.

12 . The data analytics system of claim 1 , wherein the first target system and the second target system are the same target system.

13 . A data analytics method, comprising:
in response to receiving first instructions from a first target system of a first client, the first target system separate from the data analytics system:
create a first pipeline between an append-only first data store accessible to multiple clients and a second data store, the first pipeline configured according to the first instructions to generate a client-specific data object and store the client-specific data object in the second data store;
teardown the first pipeline upon completion of storing the client-specific data object in the second data store; and
in response to receiving second instructions from a second target system of the first client, the second target system separate from the data analytics system:
create a second pipeline between the second data store and the second target system, the second pipeline configured according to the second instructions to generate query results using, at least in part, the client-specific data object and provide the query results to the second target system; and
teardown the second pipeline upon completing provision of the query results to the second target system.

14 . The method of claim 13 , wherein the first pipeline is configured according to the first instructions to generate the client-specific data object based on data from multiple distinct data sources in the append-only first data store.

15 . The method of claim 13 , wherein the first instructions comprise a declarative specification of the first pipeline; or a template personalized with metadata.

16 . The method of claim 13 , wherein the first instructions comprise a function call and the data analytics system is configurable to generate first infrastructure instructions for creating the first pipeline to implement the function call.

17 . The method of claim 13 , wherein the first pipeline is configured to generate the client-specific data object by performing at least one of validation, aggregation, filtering, classification, transformation, or coding operations.

18 . The method of claim 13 , wherein the first pipeline is configured to associate a context with the client-specific data object, the context specifying at least one of the first client, information describing the first pipeline, or security information for the client-specific data object.

19 . The method of claim 13 , wherein the second pipeline is configured according to the second instructions to provide virtualized access to an external data source and generate the query results using, at least in part, the virtualized access to the external data source.

20 . A non-transitory computer-readable medium, containing instructions that, when executed by at least one processor of a data analytics system including an append-only first data store accessible to multiple clients and a second data store, cause the data analytics system to perform operations comprising:
in response to receiving first instructions from a first target system of a first client, the first target system separate from the data analytics system:
create a first pipeline between the append-only first data store and the second data store, the first pipeline configured according to the first instructions to generate a client-specific data object and store the client-specific data object in the second data store;
teardown the first pipeline upon completion of storing the client-specific data object in the second data store; and
in response to receiving second instructions from a second target system of the first client, the second target system separate from the data analytics system:
create a second pipeline between the second data store and the second target system, the se

Assignees

• Fidelity Information Services Llc

Inventors

• Colcord Aaron David
• Mellott Kevin Richard
• Favela David Vincente
• Neong Jeffrey Chee-Keong

Classifications

• G06F9/3005

  to perform operations for flow control · CPC title

• G06F16/211

  Schema design and management · CPC title

• G06F16/2471 Primary

  Distributed queries · CPC title

• G06F16/2465

  Query processing support for facilitating data mining operations in structured databases · CPC title

• G06F21/6227

  where protection concerns the structure of data, e.g. records, types, queries · CPC title