Data processing in a hybrid cluster environment

Patent metadata

Field	Value
Publication number	US-2019278938-A1
Application number	US-201815915428-A
Country	US
Kind code	A1
Filing date	Mar 8, 2018
Priority date	Mar 8, 2018
Publication date	Sep 12, 2019
Grant date	—

Abstract

A hybrid cluster environment with a public cloud cluster having nodes storing data and a plurality of private clusters is provided, wherein each of the plurality of private clusters has nodes storing data. Registration data that indicates a customer identifier, a new private cluster, and a file transfer server is received. The new private cluster is added to the plurality of private clusters in the hybrid cluster environment. Input to design a job to process data in the hybrid cluster environment is received. It is determined that the job is to be deployed to the new private cluster. The job is deployed to the new private cluster using the file transfer server, wherein the job is executed at the new private cluster. Job status information and one or more job logs are received with the file transfer server.

First claim

1 - 7 . (canceled)

8 . A computer program product, the computer program product comprising a computer readable storage medium having program code embodied therewith, the program code executable by at least one processor to perform operations comprising:
providing a hybrid cluster environment with a public cloud cluster having nodes storing data and a plurality of private clusters, wherein each of the plurality of private clusters has nodes storing data; receiving registration data that indicates a customer identifier, a new private cluster, and a file transfer server; adding the new private cluster to the plurality of private clusters in the hybrid cluster environment; receiving input to design a job to process data in the hybrid cluster environment; determining that the job is to be deployed to the new private cluster; deploying the job to the new private cluster using the file transfer server, wherein the job is executed at the new private cluster; and
receiving job status information and one or more job logs with the file transfer server.

9 . The computer program product of claim 8 , wherein the job is deployed at any combination of: the public cloud cluster and any cluster from the plurality of private clusters.

10 . The computer program product of claim 8 , wherein the registration data identifies a master node of the new private cluster.

11 . The computer program product of claim 10 , wherein the program code is executable by the at least one processor to perform further operations comprising:
selecting a node of the nodes of the new private cluster using a load balancing proxy server of the master node of the new private cluster; executing the job to process the data at the selected node of the new private cluster while monitoring the execution to generate the job status information and the one or more job logs without moving the data out of the selected node; and returning the job status information and the one or more job logs using the file transfer server of the new private cluster.

12 . The computer program product of claim 8 , wherein a cluster management service of the hybrid cluster environment creates the job based on the input to design the job.

13 . The computer program product of claim 8 , wherein the new private cluster is one of an on-premise cluster at a customer location and a private cloud cluster.

14 . The computer program product of claim 8 , wherein a Software as a Service (SaaS) is configured to perform the operations of the computer program product.

15 . A computer system, comprising:
one or more processors, one or more computer-readable memories and one or more computer-readable, tangible storage devices; and program instructions, stored on at least one of the one or more computer-readable, tangible storage devices for execution by at least one of the one or more processors via at least one of the one or more memories, to perform operations comprising: providing a hybrid cluster environment with a public cloud cluster having nodes storing data and a plurality of private clusters, wherein each of the plurality of private clusters has nodes storing data; receiving registration data that indicates a customer identifier, a new private cluster, and a file transfer server; adding the new private cluster to the plurality of private clusters in the hybrid cluster environment; receiving input to design a job to process data in the hybrid cluster environment; determining that the job is to be deployed to the new private cluster; deploying the job to the new private cluster using the file transfer server, wherein the job is executed at the new private cluster; and
receiving job status information and one or more job logs with the file transfer server.

16 . The computer system of claim 15 , wherein the job is deployed at any combination of: the public cloud cluster and any cluster from the plurality of private clusters.

17 . The computer system of claim 15 , wherein the registration data identifies a master node of the new private cluster.

18 . The computer system of claim 17 , wherein the operations further comprise:
selecting a node of the nodes of the new private cluster using a load balancing proxy server of the master node of the new private cluster; executing the job to process the data at the selected node of the new private cluster while monitoring the execution to generate the job status information and the one or more job logs without moving the data out of the selected node; and returning the job status information and the one or more job logs using the file transfer server of the new private cluster.

19 . The computer system of claim 15 , wherein a cluster management service of the hybrid cluster environment creates the job based on the input to design the job.

20 . The computer system of claim 15 , wherein a Software as a Service (SaaS) is configured to perform operations of the computer system.

Assignees

• IBM

Inventors

• Greene Lawrence A
• Li Yong
• Pham Ryan
• Pu Xiaoyan
• Sheng Yeh-Heng

Classifications

• G06F9/5072

  Grid computing · CPC title

• H04L43/0876

  Network utilisation, e.g. volume of load or congestion level · CPC title

• H04L41/5019

  Ensuring fulfilment of SLA · CPC title

• H04L67/34

  involving the movement of software or configuration parameters  (network booting or remote initial program loading [RIPL] G06F9/4416) · CPC title

• H04L67/06

  specially adapted for file transfer, e.g. file transfer protocol [FTP] · CPC title