Distributed multiple tier multi-node serverless framework for complex analytics task execution

Patent metadata

Field	Value
Publication number	US-10719360-B2
Application number	US-201816159399-A
Country	US
Kind code	B2
Filing date	Oct 12, 2018
Priority date	Apr 11, 2018
Publication date	Jul 21, 2020
Grant date	Jul 21, 2020

Abstract

A system may support distributed multiple tier multi-node serverless analytics task execution. At a data ingestion tier, data ingestion serverless tasks may receive detail data for analytic processing. data integration serverless tasks, executing at a data integration and consolidation tier and initiated by the data ingestion serverless tasks, may sort the detail data and identify patterns within the detail data to generate grouped pre-processed data. The data integration serverless tasks may initiate partitioning serverless tasks which may divide the grouped pre-processed data into data chunks. Multi-node analytic serverless tasks at an analytic tier, at least some of which being initiated by the partitioning serverless tasks, may analyze the data chunks and generate prescriptive outputs.

First claim

What is claimed is:

1. A system including:
network interface circuitry configured to:
receive ingestion detail data from multiple data sources; and
sending first and second prescriptive outputs to a storage host server; and

tiered analytic processing circuitry configured to execute multiple tiers of serverless tasks, the multiple tiers including:
a data ingestion tier;
a data integration and consolidation tier;
a partitioning tier; and
an analytic tier;

the tiered analytic processing circuitry configured to:
establish, at the data ingestion tier, a data ingestion serverless task configured to:
access, via the network interface circuitry, the ingestion detail data; and
generate an ingestion inter-tier communication link; and
via the ingestion inter-tier communication link:
initiate a data integration serverless task;

responsive to initiation from the data ingestion serverless task, establish, at the data integration and consolidation tier, the data integration serverless task, the data integration serverless task configured to:
sort the ingestion detail data into an association grouping to generate grouped detail data;
parse the grouped detail data to identify patterns present within the grouped detail data to generate grouped pre-processed data;
generate a data integration inter-tier communication link; and
via the data integration inter-tier communication link:
initiate a partitioning serverless task;

responsive to initiation from the data integration serverless task, establish, at the partitioning tier, the partitioning serverless task, the partitioning serverless task configured to:
identify a partitioning criterion and a concurrency criterion based on a serverless task execution limit and a size of the grouped pre-processed data;
partition the grouped pre-processed data in accord with the partitioning criterion to generate a first data chunk and a second data chunk;
generate a partitioning inter-tier communication link; and
via the partitioning inter-tier communication link:
initiate a first multi-node analytic serverless task;

responsive to initiation from the partitioning serverless task, establish, at the analytic tier, the first multi-node analytic serverless task configured to:
analyze the first data chunk and at least a first portion the patterns present within the grouped detail data to generate the first prescriptive output; and
initiate a second multi-node analytic serverless task, the period based on the concurrency criterion; and

responsive to initiation from the first multi-node analytic serverless task, establish, at the analytic tier, the second multi-node analytic serverless task, the second multi-node analytic serverless task configured to analyze the second data chunk and at least a second portion the patterns present within the grouped detail data to generate the second prescriptive output.

2. The system of claim 1 , where:
the multiple tiers further include a data merge tier; and
the tiered analytic processing circuitry is further configured to:
establish, in accord with the concurrency criterion, a final multi-node analytic serverless task configured to:
generate an analytic inter-tier communication link; and
via the analytic inter-tier communication link, initiate a data merge serverless task; and

responsive to initiation from the final multi-node analytic serverless task, establish, at the data merge tier, a data merge serverless task configured to merge the first and second prescriptive outputs.

3. The system of claim 2 , where the second multi-node analytic serverless task is the final multi-node analytic serverless task.

4. The system of claim 1 , where the data integration serverless task is configured to sort the ingestion detail data into multiple association groupings based on multiple predetermined associations.

5. The system of claim 4 , where the storage host server is configured to control multiple databases each defined for a respective one of the multiple association groupings.

6. The system of claim 4 , where the multiple association groupings include one or more from among: a client association, a sensor type association, a geographic association, a tagging association, and a utility association.

7. The system of claim 1 , where the data integration serverless task is further configured to: prior to sorting the ingestion detail data, exclude ingestion detail data outside of a defined analysis window.

8. The system of claim 1 , where the serverless task execution limit includes a pre-defined serverless task timeout.

9. The system of claim 1 , where the size of the grouped pre-processed data includes a number of entries within the grouped pre-processed data.

10. The system of claim 1 , where one or more from among: the data ingestion serverless task, the data integration serverless task, and the partitioning serverless task include a series of serverless tasks.

11. The system of claim 10 , where each serverless task within the series is partially concurrent with another one of the serverless tasks in the series.

12. The system of claim 1 , where the partitioning serverless task is configured to determine the partitioning criterion and the concurrency criterion further based on a target processing completion schedule for the analytics tier.

13. The system of claim 12 , where the partitioning serverless task is configured to determine the partitioning criterion and the concurrency criterion further based on the target processing completion schedule for the analytics tier by determining a number of multi-node analytic serverless tasks with a capacity to finish processing the grouped pre-processed data prior to a completion target time.

14. The system of claim 1 , where the data ingestion serverless task is configured to pass the ingestion detail data to the data integration serverless task by providing the data integration serverless task with a storage location for the ingestion detail data.

15. A method including:
at network interface circuitry:
receiving ingestion detail data from multiple data; and
sending first and second prescriptive outputs to a storage host server; and

at tiered analytic processing circuitry configured to execute multiple tiers of serverless tasks:
establishing, at a data ingestion tier of the multiple tiers, a data ingestion serverless task configured to:
accessing, via execution of the data ingestion serverless task, the ingestion detail data;
generating, via execution of the data ingestion serverless task, an ingestion inter-tier communication link;
via the ingestion inter-tier communication link:
initiating, via execution of the data ingestion serverless task, a data integration serverless task;

responsive to initiation from the data ingestion serverless task, establishing, at a data integration and consolidation tier of the multiple tiers, the data integration serverless task;
sorting, via execution of the data integration serverless task, the ingestion detail data into an association grouping to generate grouped detail data;
parsing, via execution of the data integration serverless task, the grouped detail data to identify patterns present within the grouped detail data to generate grouped pre-processed data;
generating, via execution of the data integration serverless task, a data integration inter-tier communication link;
via the data integration inter-tier communication link:
initiating, via execution of the data integration serverless task, a partitioning serverless task; and

responsive to initiation from the data integration serverless task, establishing, at a

Assignees

• Accenture Global Solutions Ltd

Inventors

• Srinivasan Madhan Kumar
• Purushothaman Arun
• Achanta Vijaya Tapaswi

Classifications

• H04L67/60

  Scheduling or organising the servicing of application requests, e.g. requests for application data transmissions using the analysis and optimisation of the required network resources (admission control or resource allocation H04L47/70) · CPC title

• H04L67/12

  specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks · CPC title

• G06F9/4887 Primary

  involving deadlines, e.g. rate based, periodic · CPC title

• G06F16/24578

  using ranking · CPC title

• G06F16/9535

  Search customisation based on user profiles and personalisation · CPC title