Cluster job submission

Patent metadata

Field	Value
Publication number	US-11748161-B1
Application number	US-202016917616-A
Country	US
Kind code	B1
Filing date	Jun 30, 2020
Priority date	Jun 30, 2020
Publication date	Sep 5, 2023
Grant date	Sep 5, 2023

Abstract

A method and apparatus for job submission are described. In one embodiment, the jobs are submitted by a job submission service or gateway that schedules large-scale data processing jobs on remote infrastructure. In one embodiment, the method comprises: receiving a request at a proxy service from a first client, via a first network communication, to submit a first job to a cluster; and managing the first job externally to the first client, including sending a request to an orchestration system to launch an orchestration system job in a container to start the first job running on the cluster via a client process run on a job client in the container and provide state information back to the proxy service regarding the orchestration system job.

First claim

We claim:

1. A method comprising:
receiving a request at a proxy service from a first client, via a first network communication, to submit a first job to a cluster; and
managing the first job externally to the first client, including
sending, by the proxy service, a request to an orchestration system to launch an orchestration system job in a container to start a cluster job running on the cluster via a job client running on a job client server separate from the cluster and the proxy service, the container encapsulating the first job that the client requested the proxy service to submit and including the job client and a launcher wrapper that launches the job client on the job client server,
storing, in a database under control of the proxy service, job identification information to identify the first job and a mapping of the job identification information to a cluster identifier associated with the cluster job, and
receiving state information regarding the orchestration system job sent by the launcher wrapper to the proxy service for storage in the database with the job identification information and the mapping of the job identification information to the cluster identifier associated with the cluster;
receiving, by the proxy service, a kill request from the first client to stop execution of the first job; and
signaling, by the proxy service, the cluster job directly without proceeding through the orchestration system using the cluster identifier stored in memory to kill execution of the cluster job.

2. The method defined in claim 1 further comprising:
storing, under control of the proxy service, job identification information to identify the first job and one or both of status and log information regarding the first job produced by the job client in the container; and
enabling access, to the first client, to status and/or log information regarding the first job produced by the job client in the container.

3. The method defined in claim 1 further comprising:
receiving a call back from the container with information indicating the orchestration system job is running;
receiving a heartbeat messages from the container indicating the orchestration system job is running; and
receiving one or more logs emitted by the job client.

4. The method defined in claim 1 wherein sending the request to the orchestration system to launch the orchestration system job comprises launching a container that encapsulates the first job.

5. The method defined in claim 4 wherein the encapsulated job includes hooks for one or more operations for the cluster to perform.

6. The method defined in claim 1 wherein the first job is a batch job.

7. The method defined in claim 1 wherein the request is an HTTP request.

8. The method defined in claim 1 wherein the cluster is a Hadoop cluster, and wherein the orchestration system is Kubernetes.

9. The method of claim 1 further comprising:
deleting, by the proxy service, the orchestration system job in response to the kill request.

10. A system comprising:
a network interface to receive a request at a proxy service from a first client, via a first network communication, to submit a first job to a cluster;
a memory to store instructions;
one or more processors coupled to the memory and the network interface to execute the stored instructions to:
manage the first job externally to the first client, by
sending, by the proxy service, a request to an orchestration system to launch an orchestration system job in a container to start a cluster job running on the cluster via a job client running on a job client server separate from the cluster and the proxy service, the container encapsulating the first job that the client requested the proxy service to submit and including the job client and a launcher wrapper that launches the job client on the job client server,
storing, in a database under control of the proxy service, job identification information to identify job the first job and a mapping of the job identification information to a cluster identifier associated with the cluster job,
receiving state information regarding the orchestration system job sent by the launcher wrapper to the proxy service for storage in the database with the job identification information and the mapping of the job identification information to the cluster identifier associated with the cluster,
receiving, by the proxy service, a kill request from the first client to stop execution of the first job, and
signaling, by the proxy service, the cluster job directly without proceeding through the orchestration system using the cluster identifier stored in memory to kill execution of the cluster job.

11. The system defined in claim 10 wherein the one or more processors are configured to:
store, under control of the proxy service, job identification information to identify the first job and one or both of status and log information regarding the first job produced by the job client in the container; and
enable access, to the first client, to status and/or log information regarding the first job produced by the job client in the container.

12. The system defined in claim 10 wherein the network interface is operable to:
receive a call back from the container with information indicating the orchestration system job is running;
receive a heartbeat messages from the container indicating the orchestration system job is running; and
receive one or more logs emitted by the job client.

13. The system defined in claim 10 wherein the one or more processors send the request to the orchestration system to launch the orchestration system job by launching a container that encapsulates the first job.

14. The system defined in claim 13 wherein the encapsulated job includes hooks for one or more operations for the cluster to perform.

15. The system defined in claim 10 wherein the request is an HTTP request.

16. One or more non-transitory computer readable storage media having instructions stored thereupon which, when executed by a system having at least a processor and a memory therein, cause the system to perform operations comprising:
receiving a request at a proxy service from a first client, via a first network communication, to submit a first job to a cluster; and
managing the first job externally to the first client, including
sending, by the proxy service, a request to an orchestration system to launch an orchestration system job in a container to start a cluster job running on the cluster via a job client running on a job client server separate from the cluster and the proxy service, the container encapsulating the first job that the client requested the proxy service to submit and including the job client and a launcher wrapper that launches the job client on the job client server,
storing, in a database under control of the proxy service, job identification information to identify job the first job and a mapping of the job identification information to a cluster identifier associated with the cluster job,
receiving state information regarding the orchestration system job sent by the launcher wrapper to the proxy service for storage in the database with the job identification information and the mapping of the job identification information to the cluster identifier associated with the cluster,
receiving, by the proxy service, a kill request from the first client to stop execution of the first job, and
signaling, by the proxy service, the cluster job directly without proceeding through the orchestration system using the cluster identifier stored in memory to kill execut

Assignees

• Stripe Inc

Inventors

• Johnson Andrew
• Snitkovskiy Daniel
• Motoyama Marti
• Bender Jonathan

Classifications

• G06F9/5027

  the resource being a machine, e.g. CPUs, Servers, Terminals · CPC title

• G06F9/5077

  Logical partitioning of resources; Management or configuration of virtualized resources (specific details on emulation or internal functioning of virtual machines G06F9/455) · CPC title

• G06F9/5005 Primary

  to service a request · CPC title

• G06F9/455

  Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines · CPC title

• G06F9/45533

  Hypervisors; Virtual machine monitors · CPC title