Cloud-based database-less serverless framework using data foundation

Patent metadata

Field	Value
Publication number	US-11334590-B2
Application number	US-201816235815-A
Country	US
Kind code	B2
Filing date	Dec 28, 2018
Priority date	Dec 28, 2018
Publication date	May 17, 2022
Grant date	May 17, 2022

Abstract

A system may support multiple tier serverless data foundation creation to support large data set processing. At a data ingestion tier, data ingestion serverless tasks may receive source data for processing. The data integration serverless tasks may filter and group the source data into file-object stored items. Further, data integration serverless tasks may capture metadata that, when paired with the file-object stored items, establishes the data foundation. The data foundation facilitates database-like performance in data operations in a database-less system. At the processing tier, the processing serverless tasks access the data foundation by iterating across the file-object stored items to generate output-object stored items. At the directed storage tier, directed storage serverless tasks capture metadata for the output-object stored items to establish an output data foundation or prepare the output data for storage in a data warehouse.

First claim

What is claimed is:

1. A system including:
network interface circuitry configured to access a source stored item;
tiered analytic processing circuitry in data communication with the network interface circuitry, the tiered analytic processing circuitry configured to execute multiple tiers of serverless tasks, the multiple tiers including:
a data ingestion tier;
a processing tier; and
a directed storage tier;

the tiered analytic processing circuitry configured to:
establish, at the data ingestion tier, a data ingestion serverless task configured to:
create a database-less data foundation by:
for each entity identifier of one or more entity identifiers:
 filtering content within the source stored item based on the entity identifier to generate entity data;
 filtering the entity data based on a topic-associated query to generate topic-associated data;
 grouping the topic-associated data into one or more file-object stored items;
 capturing input metadata for the one or more file-object stored items based on the topic-associated query; and
 writing the input metadata to an input data foundation;

generate an ingestion inter-tier communication link; and
via the ingestion inter-tier communication link:
initiate a processing serverless task;

responsive to initiation from the data ingestion serverless task, establish, at the processing tier, the processing serverless task, the processing serverless task configured to:
access the input data foundation;
iterate across the one or more file-object stored items by:
for each of the one or more file-object stored items:
 based on the input metadata within the input data foundation for a current file-object stored item:
  determining an order for multiple computational operations for the current file-object stored item;
  executing the multiple computational operations on the current file-object stored item according to the order; and
  via the multiple computational operations, writing to one or more current output-object stored items stored among one or more output-object stored items, wherein the multiple computational operations on the current file-object stored item translate the current file-object stored item into the one or more current output-object stored items by distributing results of the multiple computational operations on the current file-object stored item into multiple corresponding output-object stored items;

generate a processing inter-tier communication link; and
via the processing inter-tier communication link:
initiate a directed storage serverless task; and

responsive to initiation from the processing serverless task, establish, at the directed storage tier, the directed storage serverless task, the directed storage serverless task configured to:
iterate across the one or more output-object stored items by:
for each of the one or more output-object stored items:
 capturing output metadata based on:
  the input metadata for the one or more file-object stored items, the one or more file-object stored items being applied to generate the output-object stored item; and
  the computational operations performed on the one or more file-object stored items applied to generate the output-object stored item; and
 writing the output metadata to an output data foundation.

2. The system of claim 1 , where the input data foundation is supported via storage within the one or more file-object stored items.

3. The system of claim 2 , where the input metadata is stored within the one or more file-object stored items.

4. The system of claim 1 , where the output data foundation is supported via storage within the one or more output-object stored items.

5. The system of claim 4 , where the output metadata is stored within the one or more output-object stored items.

6. The system of claim 1 , where the network interface circuitry is configured to send a request to a serverless task host interface to establish the data ingestion serverless task, the processing serverless task, the directed storage serverless task, or any combination thereof.

7. The system of claim 1 , where the data ingestion serverless task is configured to, via communication over the network interface circuitry, access a storage host interface to perform storage operations on the one or more file-object stored items.

8. The system of claim 1 , where grouping the topic-associated data into the one or more file-object stored items includes grouping the topic-associated data into an entity-specific file-object stored item.

9. The system of claim 1 , where the data ingestion serverless task, the processing serverless task, the directed storage serverless task, or any combination thereof includes a serially executed chain of continuity-maintaining serverless tasks.

10. The system of claim 9 , where the continuity-maintaining serverless tasks are configured to maintain continuity by maintaining an overlap parameter detailing a timing overlap for successive serverless tasks.

11. The system of claim 1 , where the data ingestion serverless task is configured to: before initiating the processing serverless task, initiate a partitioning serverless task configured to partition an analytic analysis routine into multiple chunks for completion by multiple continuity-maintaining processing serverless tasks, wherein sizes of the chunks are determined based on at least one of the following factors: execution limits of the processing serverless tasks, a duration of overlap between the processing serverless tasks, an overall size of the one or more file-object stored items, a number of all the processing serverless tasks, and complexity of the computational operations.

12. The system of claim 1 , where the processing serverless task is configured to iterate across the one or more file-object stored items further by iterating over storage identifiers for the one or more file-object stored items.

13. The system of claim 1 , where the input metadata for the one or more file-object stored items is stored in one or more other file-object stored items.

14. The system of claim 1 , where the processing serverless task is further configured to:
for each of the one or more output-object stored items, concatenate results of the multiple computational operations on the current file-object stored item with results from a previous file-object stored item also stored in the current output-object stored item.

15. A method including:
accessing, via network interface circuitry, a source stored item;
at tiered analytic processing circuitry configured to execute multiple tiers of serverless tasks:
establishing, at a data ingestion tier of the multiple tiers, a data ingestion serverless task;
via the data ingestion serverless task:
creating a database-less data foundation by:
for each entity identifier of one or more entity identifiers:
 filtering content within the source stored item based on the entity identifier to generate entity data;
 filtering the entity data based on a topic-associated query to generate topic-associated data;
 grouping the topic-associated data into one or more file-object stored items;
 capturing input metadata for the one or more file-object stored items based on the topic-associated query; and
 writing the input metadata to an input data foundation;

generating an ingestion inter-tier communication link; and
via the ingestion inter-tier communication link:
initiating a processing serverless task;

responsive to initiation from the data ingestion serverless task, establishing, at a processing tier of the multiple tiers, the processing serverless tas

Assignees

• Accenture Global Solutions Ltd

Inventors

• Srinivasan Madhan Kumar
• Purushothaman Arun
• Achanta Vijaya Tapaswi

Classifications

• G06F16/256 Primary

  in federated or virtual databases · CPC title

• G06F16/254 Primary

  Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses · CPC title

• G06F16/148

  File search processing · CPC title

• G06F16/285

  Clustering or classification · CPC title

• G06F16/13

  File access structures, e.g. distributed indices (arrangements of input from, or output to, record carriers G06F3/06) · CPC title