Fault recovery management in a cloud computing environment

Patent metadata

Field	Value
Publication number	US-10664348-B2
Application number	US-201816048259-A
Country	US
Kind code	B2
Filing date	Jul 28, 2018
Priority date	Jul 26, 2016
Publication date	May 26, 2020
Grant date	May 26, 2020

Abstract

Technologies for managing fault recovery in a cloud computing environment may be used after faults of various sizes, including faults which put total functioning capacity below subscribed capacity. Computing services have repair priorities. A fault recovery manager selects a higher priority service whose capacity is below a minimum availability, and chooses a lower priority service still above its minimal availability, and reassigns capacity from the lower priority service to the higher priority service without depriving the lower priority service of operability. Capacity reassignment continues at least until the higher priority service is at or above minimal availability, or the lower priority service is at minimal availability. Lower priority services may also be terminated entirely to free up resources for higher priority services. New deployments may be prevented until all services are at or above minimal availability. Spare capacity may be reserved against demand fluctuations or further faults.

First claim

What is claimed is:

1. A computing technology method for fault recovery management by a tenant or an infrastructure provider in a cloud computing environment having computational resources which support instances of computing services, the method comprising:
using at least one declarative statement in a recovery policy to specify at least two computing service repair priorities;
selecting a higher priority computing service which has a current number of working instances that is below a minimal availability threshold of the higher priority computing service, the higher priority computing service also having a repair priority which is among the computing service repair priorities specified in the recovery policy;
choosing a lower priority computing service which has a repair priority that is lower than the higher priority computing service's repair priority, the lower priority being among the computing service repair priorities specified in the recovery policy, the lower priority computing service also having a current number of working instances that is above a minimal availability threshold of the lower priority computing service;
reducing the current number of working instances of the chosen lower priority computing service, thereby freeing for reassignment at least a portion of at least one computational resource of the cloud computing environment;
increasing the number of working instances of the higher priority computing service, thereby reassigning at least part of the freed computational resource; and
repeating at least said reducing and increasing at least until one of the following conditions occurs: (i) the number of working instances of the higher priority computing service is no longer below the minimal availability threshold of the higher priority computing service, or (ii) further reducing would put the number of working instances of the lower priority computing service below the minimal availability threshold of the lower priority computing service.

2. The method of claim 1 , wherein the method comprises reading at least two of the following declarative statements in a recovery policy:
a statement describing what to do when a node fails;
a statement describing what to do when a rack fails;
a statement describing what to do when a fault domain fails;
a statement describing what to do when a datacenter fails;
a statement describing what to do when a datacenter group fails;
a statement describing what to do when an availability zone fails; or
a statement describing what to do when a region fails.

3. The method of claim 2 , wherein the method comprises reading at least three of the listed recovery policy declarative statements.

4. The method of claim 1 , wherein the method comprises a fault recovery manager communicating with infrastructure software rather than communicating directly with the computing services.

5. The method of claim 1 , wherein the minimal availability threshold of the higher priority computing service includes the computing service having a number of working instances which is at least one hundred.

6. The method of claim 1 , wherein the method further comprises authenticating a fault recovery manager and at least one of the computing services to one another, thereby reducing a risk of rogue behavior.

7. The method of claim 1 , wherein the method further comprises taking one or more measures to securely control updates to a fault recovery manager, thereby reducing a risk of rogue behavior.

8. The method of claim 1 , wherein the method further comprises detecting at least one of the following kinds of rogue behavior:
a transmission of a capacity reallocation command which involves falsified identification of a fault recovery manager;
a transmission of a capacity reallocation command which commands a spurious action;
a transmission of a capacity reallocation command which commands a malicious action;
a performance of a capacity reallocation command which involves falsified identification of a fault recovery manager;
a performance of a capacity reallocation command which commands a spurious action;
a performance of a capacity reallocation command which commands a malicious action;
a non-performance of an authorized capacity reallocation command; or
a tampering with capacity reallocation to favor one tenant over other tenants.

9. A computer-readable storage medium configured with executable instructions to perform a method for fault recovery management in a cloud computing environment including at least one region having computational resources which support computing services, the method comprising:
determining that a current total functioning capacity of the region is less than a total subscribed capacity of all the computing services in the region;
selecting a higher priority computing service which has a current assigned capacity that is below a minimal availability threshold of the higher priority computing service, the higher priority computing service also having a repair priority;
choosing a lower priority computing service which has a repair priority that is lower than the higher priority computing service's repair priority, the lower priority computing service also having a current assigned capacity that is above a minimal availability threshold of the lower priority computing service;
reducing the current assigned capacity of the chosen lower priority computing service, thereby freeing for reassignment at least a portion of at least one computational resource of the cloud computing environment;
increasing the assigned capacity of the higher priority computing service, thereby reassigning at least part of the freed computational resource; and
repeating at least said reducing and increasing at least until one of the following conditions occurs: (i) the assigned capacity of the higher priority computing service is no longer below the minimal availability threshold of the higher priority computing service, (ii) the assigned capacity of the higher priority computing service is at least a subscribed capacity of the higher priority computing service, or (iii) further reducing would put the assigned capacity of the lower priority computing service below the minimal availability threshold of the lower priority computing service.

10. The computer-readable storage medium of claim 9 , wherein the method further comprises avoiding application latency from service reductions by favoring a specific hosting location which is declared in a service definition recovery policy.

11. The computer-readable storage medium of claim 9 , wherein at least one of the computing services is provided as part of an infrastructure-as-a-service offering.

12. The computer-readable storage medium of claim 9 , wherein at least one of the computing services is provided as part of a platform-as-a-service offering.

13. The computer-readable storage medium of claim 9 , wherein at least one of the computing services is provided as part of a software-as-a-service offering.

14. The computer-readable storage medium of claim 9 , wherein said selecting, choosing, reducing and increasing are repeated until each computing service has at least a minimal availability, in that for each computing service in the cloud computing environment the assigned capacity of the computing service is not below the minimal availability threshold of the computing service, and wherein the cloud computing environment has unassigned computational resources remaining after the assigned capacity of each non-terminated and non-suspended computing service is not below the minimal availability threshold of the computing service, and the method furthe

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Johnson James E
• Russinovich Mark

Classifications

• H04L67/1008 Primary

  based on parameters of servers, e.g. available memory or workload (monitoring of computer activity G06F11/30) · CPC title

• G06F11/3442

  for planning or managing the needed capacity · CPC title

• G06F2201/805

  Real-time · CPC title

• G06F2201/85

  Active fault masking without idle spares · CPC title

• G06F11/142 Primary

  Reconfiguring to eliminate the error (group management mechanisms in a peer-to-peer network H04L67/1044) · CPC title