Remotely managing execution of jobs in a cluster computing framework

Patent metadata

Field	Value
Publication number	US-11847502-B2
Application number	US-201916717234-A
Country	US
Kind code	B2
Filing date	Dec 17, 2019
Priority date	Feb 14, 2018
Publication date	Dec 19, 2023
Grant date	Dec 19, 2023

Abstract

A device, that provides serverless computing, receives a request to execute multiple jobs, and determines criteria for each of the plurality of jobs, wherein the criteria for each of the multiple jobs includes at least one of job posting criteria, job validation criteria, job retry criteria, or a disaster recovery criteria. The device stores information associated with the multiple jobs in a repository, wherein the information associated with the multiple jobs includes the criteria for each of the multiple jobs. The device provides a particular job, of the multiple jobs, to a cluster computing framework for execution, determines modified criteria for the particular job, and provides the modified criteria for the particular job to the cluster computing framework. The device receives, from the cluster computing framework, information indicating that execution of the particular job is complete, and validates a success of completion of the execution of the particular job.

First claim

What is claimed is:

1. A method, comprising:
receiving, by a device, a request for execution of a plurality of jobs;
determining, by the device and based on receiving the request, execution criteria, for a particular job of the plurality of jobs, and retry criteria for the particular job,
wherein the execution criteria indicates that the particular job is to be executed in parallel with a second job of the plurality of jobs;

providing, by the device, to a first master device of a first cluster computing framework different from the device, a request for execution of the particular job according to the execution criteria,
wherein the first cluster computing framework comprises the first master device and one or more first executor devices to execute the particular job according to the execution criteria;

receiving, by the device, from the first master device, and based on providing the request, information associated with the one or more first executor devices failing to execute at least one of the particular job or the second job; and
sending, by the device, to the first master device, and based on the retry criteria and the information associated with the one or more first executor devices failing to execute the at least one of the particular job or the second job an instruction to re-route the particular job, and either the execution criteria or modified execution criteria, to a second master device, of a second cluster computing framework different from the device, for execution of the particular job according to the execution criteria or the modified execution criteria,
wherein the second cluster computing framework comprises the second master device and one or more second executor devices to execute of the particular job according to the execution criteria or the modified execution criteria.

2. The method of claim 1 , further comprising:
causing a job stack to be created based on the plurality of jobs, and
wherein the request is sent based on causing the job stack to be created.

3. The method of claim 1 , further comprising:
receiving, based on sending the instruction, information indicating that a first execution, of the particular job, and a second execution, of the second job, are complete; and
analyzing, based on receiving the information indicating that the first execution, of the particular job, and the second execution, of the second job, are complete, results of the first execution and the second execution.

4. The method of claim 3 , wherein analyzing the results of the first execution and the second execution comprises:
determining a first elapsed time, for the first execution of the particular job, and a second elapsed time for the second execution of the second job; and
comparing, based on determining the first elapsed time and the second elapsed time, the first execution, of the particular job, to the second execution of the second job.

5. The method of claim 3 , further comprising:
causing a job stack to be deleted based on analyzing the results of the first execution and the second execution.

6. The method of claim 1 , further comprising:
collecting performance metrics associated with the plurality of jobs; and
comparing, based on collecting the performance metrics, execution performance of the plurality of jobs.

7. The method of claim 1 , wherein the instruction is to re-route the particular job and the modified execution criteria to the second master device of the second cluster computing framework.

8. A device, comprising:
one or more memories; and
one or more processors coupled to the one or more memories, configured to:
receive a request for execution of a plurality of jobs;
determine, based on receiving the request, execution criteria, for a particular job of the plurality of jobs, and retry criteria for the particular job,
wherein the execution criteria indicates that the particular job is to be executed in parallel with a second job of the plurality of jobs;

provide, to a first master device of a first cluster computing framework different from the device, a request for execution of the particular job according to the execution criteria,
wherein the first cluster computing framework comprises the first master device and one or more first executor devices to execute the particular job according to the execution criteria;

receive, from the first master device and based on providing the request, information associated with the one or more first executor devices failing to execute at least one of the particular job or the second job; and
send, to the first master device and based on the retry criteria and the information associated with the one or more first executor devices failing to execute the at least one of the particular job or the second job an instruction to re-route the particular job, and either the execution criteria or modified execution criteria, to a second master device, of a second cluster computing framework different from the device, for execution of the particular job according to the execution criteria or the modified execution criteria,
wherein the second cluster computing framework comprises the second master device and one or more second executor devices to execute the particular job according to the execution criteria or the modified execution criteria.

9. The device of claim 8 , wherein the one or more processors are further configured to:
cause a job stack to be created based on the plurality of jobs, and
wherein the request is sent based on causing the job stack to be created.

10. The device of claim 8 , wherein the one or more processors are further configured to:
receive, based on sending the instruction, information indicating that a first execution, of the particular job, and a second execution, of the second job, are complete; and
analyze, based on receiving the information indicating that the first execution, of the particular job, and the second execution, of the second job, are complete, results of the first execution and the second execution.

11. The device of claim 10 , wherein the one or more processors, to analyze the results of the first execution and the second execution, are configured to:
determine a first elapsed time, for the first execution of the particular job, and a second elapsed time for the second execution of the second job; and
compare, based on determining the first elapsed time and the second elapsed time, the first execution, of the particular job, to the second execution of the second job.

12. The device of claim 10 , wherein the one or more processors are further configured to:
cause a job stack to be deleted based on analyzing the results of the first execution and the second execution.

13. The device of claim 8 , wherein the one or more processors are further configured to:
collect performance metrics associated with the plurality of jobs; and
compare, based on collecting the performance metrics, execution performance of the plurality of jobs.

14. The device of claim 8 , wherein the instruction is to re-route the particular job and the modified execution criteria to the second master device of the second cluster computing framework.

15. A non-transitory computer-readable medium storing instructions, the instructions comprising:
one or more instructions that, when executed by one or more processors of a device, cause the one or more processors to:
receive a request for execution of a plurality of jobs;
determine, based on receiving the request, execution criteria, for a particular job of the plurality of jobs, and retry criteria for the particular job,
wherein the execut

Assignees

• Capital One Services Llc

Inventors

• Kumar Ashwini
• Kattamuri Lakshmi Narasimha Sarma

Classifications

• G06F9/5066 Primary

  Algorithms for mapping a plurality of inter-dependent sub-tasks onto a plurality of physical CPUs (mappping at compile time, see G06F8/451) · CPC title

• G06F9/448

  Execution paradigms, e.g. implementations of programming paradigms · CPC title

• G06F9/5027 Primary

  the resource being a machine, e.g. CPUs, Servers, Terminals · CPC title

• G06F9/5038

  considering the execution order of a plurality of tasks, e.g. taking priority or time dependency constraints into consideration (scheduling strategies G06F9/4881 and subgroups) · CPC title

• G06F9/5072

  Grid computing · CPC title