Server and cloud computing resource optimization method thereof for cloud big data computing architecture

Patent metadata

Field	Value
Publication number	US-10460241-B2
Application number	US-201615372348-A
Country	US
Kind code	B2
Filing date	Dec 7, 2016
Priority date	Nov 23, 2016
Publication date	Oct 29, 2019
Grant date	Oct 29, 2019

Abstract

A server and a cloud computing resource optimization method thereof for big data cloud computing architecture are provided. The server runs a dynamic scaling system to perform the following operations: receiving a task message; executing a profiling procedure to generate a profile based on an to-be-executed task recorded in the task message; executing a classifying procedure to determine a task classification of the to-be-executed task; executing a prediction procedure to obtain a plurality of predicted execution times corresponding to a plurality of computing node numbers, a computing node type and a system parameter of the to-be-executed task; executing an optimization procedure to determine a practical computing node number of the to-be-executed task; and transmitting an optimization output message to a management server to make the management server allocate at least one data computing system to execute a program file of the to-be-executed task.

First claim

What is claimed is:

1. A server for big data cloud computing architecture, comprising:
a transceiver connected to a network; and
a processor electrically connected to the transceiver, being configured to run a dynamic scaling system to execute the following operations:
receiving a task message via the transceiver, the task message recording a user-defined attribute, a program file and a plurality of data files of a to-be-executed task, the program file and the data files being stored in a data storage system which runs in a data storage server assembly in the big data cloud computing architecture;
executing a profiling procedure that comprises the following steps of:
sampling the data files recorded in the task message to choose a plurality of sampled data files and to generate an advance execution message which records the sampled data files and the program file of the to-be-executed task;
transmitting the advance execution message to a big data computing server assembly of the big data cloud computing architecture via the transceiver so that a sampled data computing system running in the big data computing server assembly executes the program file on the sampled data files and generates a profile according to an execution result of the program file; and
receiving the profile from the big data computing server assembly via the transceiver;

executing a classifying procedure that comprises the following step of:
based on a classification model, comparing the profile with the classification model to determine a task classification of the to-be-executed task;

executing a prediction procedure that comprises the following steps of:
based on a computing node type and a system parameter recorded in a prediction sub-model corresponding to the task classification in a prediction model, assigning the computing node type and the system parameter to the to-be-executed task; and
based on the prediction sub-model, generating an execution time prediction curve of the to-be-executed task according to the task classification and the profile to obtain a plurality of predicted execution times of the to-be-executed task corresponding to a plurality of computing node numbers, the computing node numbers corresponding to the predicted execution times in one-to-one correspondence;

executing an optimization procedure that comprises the following steps of:
determining whether there is at least one additional task;
when the at least one additional task exists, determining a practical computing node number of the to-be-executed task according to the user-defined attribute and the predicted execution times of the to-be-executed task, an additional user-defined attribute and a plurality of additional predicted execution times of each of the at least one additional task, and a maximum computing resource of the big data computing server assembly;
generating an optimization output message, which records the program file, the data files, the practical computing node number, the computing node type and the system parameter of the to-be-executed task; and
transmitting the optimization output message to a management server of the big data cloud computing architecture via the transceiver so that a management system run by the management server allocates at least one data computing system running in the big data computing server assembly to execute the program file on the data files of the to-be-executed task according to the optimization output message, wherein the number of the at least one data computing system is equal to the practical computing node number.

2. The server of claim 1 , wherein the processor further executes a monitoring procedure that comprises the following step of:
updating the execution time prediction curve of the to-be-executed task when the at least one data computing system is executing the program file of the to-be-executed task.

3. The server of claim 1 , wherein the processor further determines a file size of the data files of the to-be-executed task before executing the profiling procedure, and executes the profiling procedure if the file size of the data files is greater than a threshold.

4. The server of claim 1 , wherein the user-defined attribute comprises a task execution priority value, a task deadline time, a minimum computing node number and a maximum computing node number, and the optimization procedure further comprises the following step of:
calculating a task weight value of the to-be-executed task according to the following formula:

V
=

EP
×

D

D
-
WT

where V is the task weight value, EP is the task execution priority value, D is the task deadline time of the to-be-executed task, and WT is an execution waiting time.

5. The server of claim 4 , wherein the processor determines the practical computing node number of the to-be-executed task in the optimization procedure according to the task weight value and the predicted execution times of the to-be-executed task, an additional task weight value and the additional predicted execution times of each of the at least one additional task, and the maximum computing resource of the big data computing server assembly.

6. The server of claim 5 , wherein the optimization procedure further comprises the following steps of:
calculating a plurality of ratios between the task weight value of the to-be-executed task and each of the predicted execution times;
calculating a plurality of additional ratios between the additional task weight value and each of the additional predicted execution times for each of the at least one additional task;
calculating a plurality of sum values between any of the ratios and any of the additional ratios of each of the at least one additional task;
selecting the maximum sum value from among the sum values; and
setting the practical computing node number to the computing node number which corresponds to the predicted execution time of the ratio contributing to the maximum sum value.

7. The server of claim 6 , wherein the optimization output message further records an additional practical computing node number of each of the at least one additional task, and the optimization procedure further comprises the following step of:
for each of the at least one additional task, setting the additional practical computing node number to the additional computing node number which corresponds to the additional predicted execution time of the additional ratio contributing to the maximum sum value.

8. The server of claim 1 , wherein the data storage server assembly further runs a dynamic scaling data storage system, and the dynamic scaling data storage system is configured to store the sampled data files, the profile, the classification model and the prediction model of the to-be-executed task.

9. The server of claim 1 , wherein the profile of the to-be-executed task comprises program file parse data, a sampled task log and a system resource usage record.

10. The server of claim 1 , wherein the classification model has a plurality of standard task classifications, and the processor further uses a clustering algorithm to modify the standard task classifications according to a plurality of historical profiles of a plurality of already executed tasks.

11. The server of claim 1 , further comprising storage configured to store the classification model and the prediction m

Assignees

• Inst Information Ind

Inventors

• Chou Jerry Chi-Yuan
• Lu Shih-Yu
• Chen Chen-Chun
• Lin Chan-Yi
• Lu Hsin-Tse

Classifications

• G06N7/01

  Probabilistic graphical models, e.g. probabilistic networks · CPC title

• H04L67/10

  in which an application is distributed across nodes in the network (software deployment G06F8/60; multiprogramming arrangements G06F9/46) · CPC title

• G06F9/5027 Primary

  the resource being a machine, e.g. CPUs, Servers, Terminals · CPC title

• G06N5/02 Primary

  Knowledge representation; Symbolic representation · CPC title

• G06N20/00

  Machine learning · CPC title