Proactive failure recovery model for distributed computing using a checkpoint frequency determined by a MTBF threshold

Patent metadata

Field	Value
Publication number	US-9348710-B2
Application number	US-201414445369-A
Country	US
Kind code	B2
Filing date	Jul 29, 2014
Priority date	Jul 29, 2014
Publication date	May 24, 2016
Grant date	May 24, 2016

Abstract

This disclosure generally describes methods and systems, including computer-implemented methods, computer-program products, and computer systems, for providing a proactive failure recovery model for distributed computing. One computer-implemented method includes building a virtual tree-like computing structure of a plurality of computing nodes, for each computing node of the virtual tree-like computing structure, performing, by a hardware processor, a node failure prediction model to calculate a mean time between failure (MTBF) associated with the computing node, determining whether to perform a checkpoint of the computing node based on a comparison between the calculated MTBF and a maximum and minimum threshold, migrating a process from the computing node to a different computing node acting as a recovery node, and resuming execution of the process on the different computing node.

First claim

What is claimed is:

1. A computer-implemented method, comprising:
building a virtual tree-like computing structure of a plurality of computing nodes;
for each computing node of the virtual tree-like computing structure, performing, by a hardware processor, a node failure prediction model to calculate a mean time between failure (MTBF) associated with the computing node;
determining whether to perform a checkpoint of the computing node based on a comparison between the calculated MTBF and a maximum and minimum threshold;
migrating a process from the computing node to a different computing node acting as a recovery node; and
resuming execution of the process on the different computing node.

2. The method of claim 1 , further comprising:
collecting at least a computing power and node location parameter for each computing node;
dividing the computing nodes into collections based on their node location parameter; and
sorting the computing nodes within each collection based on the computing power parameter.

3. The method of claim 2 , further comprising:
identifying an upper-limit and lower-limit threshold used to determine levels of the sorted computing nodes;
sorting the computing nodes within each collection into horizontal levels based on the computing power parameter and the identified upper-limit and lower-limit thresholds;
recording the horizontal level placement and a vertical placement into a node-record-information table associated with each computing node; and
populating each node-record-information table with a designated recovery node.

4. The method of claim 3 , wherein the upper-limit and lower-limit thresholds are determined from a cross plot of the collected computing power and node location parameters for each computing node and the vertical placement is determined based at least on the node location parameter for each computing node.

5. The method of claim 1 , wherein the MTBF is calculated based at least upon a network or data storage failure.

6. The method of claim 1 , further comprising:
creating a checkpoint when the MTBF of the computing node is less than the minimum threshold; and
updating the minimum threshold associated with the computing node to equal the MTBF.

7. The method of claim 6 , further comprising:
determining that a failure of the computing node has occurred; and
using the last checkpoint taken for the computing node as a process state.

8. A non-transitory, computer-readable medium storing computer-readable instructions, the instructions executable by a computer and configured to:
build a virtual tree-like computing structure of a plurality of computing nodes;
for each computing node of the virtual tree-like computing structure, perform a node failure prediction model to calculate a mean time between failure (MTBF) associated with the computing node;
determine whether to perform a checkpoint of the computing node based on a comparison between the calculated MTBF and a maximum and minimum threshold;
migrate a process from the computing node to a different computing node acting as a recovery node; and
resume execution of the process on the different computing node.

9. The medium of claim 8 , further including instructions to:
collect at least a computing power and node location parameter for each computing node;
divide the computing nodes into collections based on their node location parameter; and
sort the computing nodes within each collection based on the computing power parameter.

10. The medium of claim 9 , further including instructions to:
identify an upper-limit and lower-limit threshold used to determine levels of the sorted computing nodes;
sort the computing nodes within each collection into horizontal levels based on the computing power parameter and the identified upper-limit and lower-limit thresholds;
record the horizontal level placement and a vertical placement into a node-record-information table associated with each computing node; and
populate each node-record-information table with a designated recovery node.

11. The medium of claim 10 , wherein the upper-limit and lower-limit thresholds are determined from a cross plot of the collected computing power and node location parameters for each computing node and the vertical placement is determined based at least on the node location parameter for each computing node.

12. The medium of claim 8 , wherein the MTBF is calculated based at least upon a network or data storage failure.

13. The medium of claim 8 , further including instructions to:
create a checkpoint when the MTBF of the computing node is less than the minimum threshold; and
update the minimum threshold associated with the computing node to equal the MTBF.

14. The medium of claim 13 , further including instructions to:
determine that a failure of the computing node has occurred; and
use the last checkpoint taken for the computing node as a process state.

15. A computer system, comprising:
at least one hardware processor interoperably coupled with a memory storage and configured to:
build a virtual tree-like computing structure of a plurality of computing nodes;
for each computing node of the virtual tree-like computing structure, perform a node failure prediction model to calculate a mean time between failure (MTBF) associated with the computing node;
determine whether to perform a checkpoint of the computing node based on a comparison between the calculated MTBF and a maximum and minimum threshold;
migrate a process from the computing node to a different computing node acting as a recovery node; and
resume execution of the process on the different computing node.

16. The system of claim 15 , further configured to:
collect at least a computing power and node location parameter for each computing node;
divide the computing nodes into collections based on their node location parameter; and
sort the computing nodes within each collection based on the computing power parameter.

17. The system of claim 16 , further configured to:
identify an upper-limit and lower-limit threshold used to determine levels of the sorted computing nodes;
sort the computing nodes within each collection into horizontal levels based on the computing power parameter and the identified upper-limit and lower-limit thresholds;
record the horizontal level placement and a vertical placement into a node-record-information table associated with each computing node; and
populate each node-record-information table with a designated recovery node.

18. The system of claim 17 , wherein the upper-limit and lower-limit thresholds are determined from a cross plot of the collected computing power and node location parameters for each computing node and the vertical placement is determined based at least on the node location parameter for each computing node.

19. The system of claim 15 , wherein the MTBF is calculated based at least upon a network or data storage failure.

20. The system of claim 15 , further configured to:
create a checkpoint when the MTBF of the computing node is less than the minimum threshold; and
update the minimum threshold associated with the computing node to equal the MTBF;
determine that a failure of the computing node has occurred; and
use the last checkpoint taken for the computing node as a process state.

Assignees

• Saudi Arabian Oil Co

Inventors

• Al-Wahabi Khalid S

Classifications

• G06F11/1438

  Restarting or rejuvenating · CPC title

• G06F11/1407

  Checkpointing the instruction stream · CPC title

• G06F11/203 Primary

  using migration · CPC title

• G06F11/0721

  within a central processing unit [CPU] · CPC title

• G06F11/1471

  involving logging of persistent data for recovery · CPC title