Data processing in a hybrid cluster environment

Patent metadata

Field	Value
Publication number	US-2019318123-A1
Application number	US-201916453908-A
Country	US
Kind code	A1
Filing date	Jun 26, 2019
Priority date	Mar 8, 2018
Publication date	Oct 17, 2019
Grant date	—

Abstract

A hybrid cluster environment with a public cloud cluster having nodes storing data and a plurality of private clusters is provided, wherein each of the plurality of private clusters has nodes storing data. Registration data that indicates a customer identifier, a new private cluster, and a file transfer server is received. The new private cluster is added to the plurality of private clusters in the hybrid cluster environment. Input to design a job to process data in the hybrid cluster environment is received. It is determined that the job is to be deployed to the new private cluster. The job is deployed to the new private cluster using the file transfer server, wherein the job is executed at the new private cluster. Job status information and one or more job logs are received with the file transfer server.

First claim

1 . A computer-implemented method, comprising operations for:
providing a hybrid cluster environment with a public cloud cluster having nodes storing data and a plurality of private clusters, wherein each of the plurality of private clusters has nodes storing data; receiving registration data that indicates a customer identifier, a new private cluster, and a file transfer server; adding the new private cluster to the plurality of private clusters in the hybrid cluster environment; receiving input to design a job to process data in the hybrid cluster environment; determining that the job is to be deployed to the new private cluster;
deploying the job to the new private cluster using the file transfer server, wherein the job is executed at the new private cluster; and
receiving job status information and one or more job logs with the file transfer server.

2 . The computer-implemented method of claim 1 , wherein the job is deployed at any combination of: the public cloud cluster and any cluster from the plurality of private clusters.

3 . The computer-implemented method of claim 1 , wherein the registration data identifies a master node of the new private cluster.

4 . The computer-implemented method of claim 3 , further comprising operations for:
selecting a node of the nodes of the new private cluster using a load balancing proxy server of the master node of the new private cluster; executing the job to process the data at the selected node of the new private cluster while monitoring the execution to generate the job status information and the one or more job logs without moving the data out of the selected node; and returning the job status information and the one or more job logs using the file transfer server of the new private cluster.

5 . The computer-implemented method of claim 1 , wherein a cluster management service of the hybrid cluster environment creates the job based on the input to design the job.

6 . The computer-implemented method of claim 1 , wherein the new private cluster is one of an on-premise cluster at a customer location and a private cloud cluster.

7 . The computer-implemented method of claim 1 , wherein a Software as a Service (SaaS) is configured to perform the operations of the method.

8 - 20 . (canceled)

Assignees

• IBM

Inventors

• Greene Lawrence A
• Li Yong
• Pham Ryan
• Pu Xiaoyan
• Sheng Yeh-Heng

Classifications

• G06F9/5072

  Grid computing · CPC title

• H04L41/5019

  Ensuring fulfilment of SLA · CPC title

• H04L43/0876

  Network utilisation, e.g. volume of load or congestion level · CPC title

• H04L67/1004

  Server selection for load balancing · CPC title

• H04L67/34

  involving the movement of software or configuration parameters  (network booting or remote initial program loading [RIPL] G06F9/4416) · CPC title