Automated scaling of multi-tier applications using reinforced learning

Patent metadata

Field	Value
Publication number	US-9412075-B2
Application number	US-201313975239-A
Country	US
Kind code	B2
Filing date	Aug 23, 2013
Priority date	Aug 23, 2013
Publication date	Aug 9, 2016
Grant date	Aug 9, 2016

Abstract

A module and method for automatically scaling a multi-tier application, wherein each tier of the multi-tier application is supported by at least one virtual machine, selects one of reinforced learning and heuristic operation based on a policy to recommend a scaling action from a current state of the multi-tier application. If reinforced learning is selected, the reinforced learning is applied to select the scaling action from a plurality of possible actions for the multi-tier application in the current state. If heuristic operation is selected, the heuristic operation is applied to select the scaling action using a plurality of defined heuristics.

First claim

What is claimed is:

1. A method for automatically scaling a multi-tier application, the method comprising:
receiving operational metrics of the multi-tier application from virtual machines executing the multi-tier application, wherein each tier of the multi-tier application is supported by at least one virtual machine;
selecting one of reinforced learning and heuristic operation to recommend a scaling action from a current state of the multi-tier application, the current state of the multi-tier application being dependent on the operational metrics;
if the reinforced learning is selected, applying the reinforced learning to select the scaling action from a plurality of possible actions for the multi-tier application in the current state; and
if the heuristic operation is selected, applying the heuristic operation to select the scaling action using a plurality of defined heuristics.

2. The method of claim 1 , wherein selecting one of the reinforced learning and the heuristic operation includes selecting of one of the reinforced learning and the heuristic operation using probabilities based on a ratio between the number of scaling actions that have been explored and the number of possible scaling actions from the current state.

3. The method of claim 1 , wherein applying the reinforced learning includes applying Q learning to select the scaling action using a Q table with Q values, each of the Q values corresponding to a combination of a particular state and a particular scaling action.

4. The method of claim 3 , wherein applying the Q learning includes updating a Q value in the Q table for the current state of the multi-tier application and the selected scaling action using an action-value function.

5. The method of claim 4 , wherein updating the Q value includes computing a reward score for the current state of the multi-tier application and the selected scaling action using a reward function, and applying the reward score to the action-reward function.

6. The method of claim 5 , wherein the reward function includes two concave functions that meet at a predefined application performance level, the reward function providing reward scores that decrease as the performance of the multi-tier application moves further away from the predefined application performance level.

7. The method of claim 1 , wherein applying the heuristic operation includes selecting one of no-change policy, scale-up policy and scale-down policy based on latency of the multi-tier application to process requests.

8. The method of claim 7 , wherein selecting one of the no-change policy, the scale-up policy and the scale-down policy further includes selecting a tier of the multi-tier application to scale based on a history window defined by a number of scaling actions already taken.

9. The method of claim 1 , further comprising grouping states of the multi-tier application into clusters using decision tree classification based on a splitting variable.

10. The method of claim 1 , wherein the operational metrics include resource utilization metrics and application performance metrics of the multi-tier application.

11. An automatic scaling module for a multi-tier application executed on a distributed computer system, wherein each tier of the multi-tier application is supported by at least one virtual machine, the automatic scaling module comprising:
a selector configured to select one of reinforced learning and heuristic operation to recommend a scaling action from a current state of the multi-tier application;
a reinforced learning module configured to apply the reinforced learning to select the scaling action from a plurality of possible actions for the multi-tier application in the current state when the reinforced learning is selected; and
a heuristic operation module configured to apply the heuristic operation to select the scaling action using a plurality of defined heuristics when the heuristic operation is selected.

12. The automatic scaling module of claim 11 , wherein the selector is configured to select one of the reinforced learning and the heuristic operation using probabilities based on a ratio between the number of scaling actions that have been explored and the number of possible scaling actions from the current state.

13. The automatic scaling module of claim 11 , wherein the reinforced learning module is configured to select the scaling action using a Q table with Q values, each of the Q values corresponding to a combination of a particular state and a particular scaling action.

14. The automatic scaling module of claim 13 , wherein the reinforced learning module is further configured update a Q value in the Q table for the current state of the multi-tier application and the selected scaling action using an action-value function.

15. The automatic scaling module of claim 14 , wherein the reinforced learning module is configured compute a reward score for the current state of the multi-tier application and the selected scaling action using a reward function, and applying the reward score to the action-reward function to update the Q value.

16. The automatic scaling module of claim 15 , wherein the reward function includes two concave functions that meet at a predefined application performance level, the reward function providing reward scores that decrease as the performance of the multi-tier application moves further away from the predefined application performance level.

17. The automatic scaling module of claim 11 , wherein the heuristic operation module is configured to selected one of no-change policy, scale-up policy and scale-down policy based on latency of the multi-tier application to process requests.

18. The automatic scaling module of claim 17 , wherein the heuristic operation module is further configured to select a tier of the multi-tier application to scale based on a history window defined by a number of scaling actions already taken.

19. The automatic scaling module of claim 11 , further comprising a state classifier configured to group states of the multi-tier application into clusters using decision tree classification based on a splitting variable.

20. The automatic scaling module of claim 11 , further comprising a monitoring server configured to receiving operational metrics of the multi-tier application from the virtual machines executing the multi-tier application, wherein at least some of the operational metrics are used to define the state of the multi-tier application and wherein the operational metrics include resource utilization metrics and application performance metrics of the multi-tier application.

21. A computer-readable storage medium containing program instructions for automatically scaling a multi-tier application, wherein execution of the program instructions by one or more processors causes the one or more processors to perform steps comprising:
receiving operational metrics of the multi-tier application from virtual machines executing the multi-tier application, wherein each tier of the multi-tier application is supported by at least one virtual machine;
based on a policy related to the operational metrics, selecting one of reinforced learning and heuristic operation to recommend a scaling action from a current state of the multi-tier application, the current state of the multi-tier application being dependent on the operational metrics;
if the reinforced learning is selected, applying the reinforced learning to select the scaling action from a plurality of possible actions for the mul

Assignees

• Vmware Inc

Inventors

• Padala Pradeep
• Parikh Aashish
• Jiang Zhefu

Classifications

• G06N99/005 Primary

  Physics · mapped topic

• G06N20/00 Primary

  Machine learning · CPC title

• G06F9/45558 Primary

  Hypervisor-specific management and integration aspects · CPC title

• G06F3/0613

  in relation to throughput · CPC title

• G06F3/0631

  by allocating resources to storage systems · CPC title