Auto-didacted hierarchical failure recovery for remote access controllers

Patent metadata

Field	Value
Publication number	US-2016253246-A1
Application number	US-201514632685-A
Country	US
Kind code	A1
Filing date	Feb 26, 2015
Priority date	Feb 26, 2015
Publication date	Sep 1, 2016
Grant date	—

Abstract

A method and an information handling system (IHS) provide hierarchical failure recovery for firmware components of the information handling system. According to one aspect, a hierarchical recovery and learning module (HRLM) detects firmware component failure. The HRLM initiates a hierarchical failure recovery by executing recovery sequences from a set of hierarchically ordered recovery sequences. The HRLM determines whether the detected failure was corrected by executing the recovery sequence. If the HRLM further determines that the failure is not corrected by executing the set of hierarchically ordered recovery sequences, the HRLM dynamically generates a new recovery sequence(s) using commands selected from at least one of the previously executed recovery sequences and executes the new recovery sequence(s). If the failure is corrected by a new recovery sequence, the HRLM forwards the particular recovery sequence to a customer support site for use by other systems in addressing similar or identical failures.

First claim

What is claimed is:

1 . A method of hierarchical self-healing and auto-didactic recovery for firmware components of an information handling system, the method comprising:
detecting a failure of a firmware component in the information handling system; initiating a hierarchical failure recovery by executing at least one recovery sequence from a set of hierarchically ordered recovery sequences, corresponding to the detected failure; following execution of a recovery sequence, performing, before executing a next recovery sequence, an evaluating to determine whether the detected failure was corrected by executing the recovery sequence; in response to the failure not being corrected by executing the set of hierarchically ordered recovery sequences, dynamically generating at least one new recovery sequence using commands selected from at least one of the previously executed recovery sequences in order to extend the set of hierarchically ordered recovery sequences; and executing the at least one new recovery sequence.

2 . The method of claim 1 , further comprising:
updating, based on performed evaluations, one or more metrics respectively associated with a success rate of the one or more executed recovery sequences relative to each other in resolving the detected failure, wherein the one or more metrics are utilized to provide an updated set of hierarchically ordered recovery sequences appropriate for addressing a future failure.

3 . The method of claim 1 , further comprising:
in response to the failure being corrected by executing the recovery sequence, dynamically initiating a learning mechanism that: records information about a respective failure and at least one corresponding recovery sequence; enables retrieval of the recorded information for report collection; and forwards the particular recovery sequence to a customer support site which subsequently provides the recovery sequence for download to other systems.

4 . The method of claim 1 , further comprising:
retrieving the set of hierarchically ordered recovery sequences; selecting a portion of commands from the at least one of the previously executed recovery sequences to create the at least one new sequence; selectively ordering the executable commands corresponding to each of the at least one new sequence, respectively, to provide at least one corresponding series of executable commands; and wherein each of the at least one series of executable commands differ from any other series of executable commands with respect to at least one of (a) constituent executable commands and (b) an ordering of the constituent executable commands, wherein each of the at least one series of executable commands is a corresponding new recovery sequence.

5 . The method of claim 4 , further comprising:
respectively ordering selected portions of the executable commands by applying one or more metrics associated with at least one of success rates of various recovery sequences having executable commands, a type of failure of the firmware component, and a priority associated with the identified failure.

6 . The method of claim 1 , further comprising:
verifying whether the detected failure has been corrected by executing a new recovery sequence; and in response to verifying that the detected failure has been corrected:
storing information associating the specific firmware failure with the new recovery sequence that corrected the failure; and
publishing an association of the detected failure and the new recovery sequence to a support site to share via download with other information handling systems.

7 . The method of claim 1 , wherein the recovery sequences are hierarchically ordered based on inter-layer system component dependencies, where various layers represent respective hierarchical levels.

8 . The method of claim 3 , further comprising:
in response to establishing that the new recovery sequence, having a specific series and group of commands, corrects the failure, recording information identifying the new recovery sequence associated with the detected failure of the firmware component, and which includes a type of failure of the firmware component and a hierarchical level of the correction.

9 . The method of claim 1 , wherein:
detecting a failure of a firmware component further comprises inserting at least one software assertion in the firmware component, wherein the software assertion can determine whether the firmware component is working properly; and once a failure is detected, the method further comprises:
identifying whether at least one hierarchical failure recovery sequence is available for that detected failure; and
generating an output indicating that the failure has been detected.

10 . The method of claim 9 , wherein generating an output indicating that a failure has been detected comprises one or more of:
displaying a message on a screen of the information handling system indicating that a failure has been detected and the information handling system is entering in the self-healing mode; and activating an LED that indicates that the information handling system is in the self-healing mode.

11 . The method of claim 10 , further comprising:
in response to not being able to identify a hierarchical failure recovery sequence for the detected failure, performing one or more of: prompting a user to manually select a hierarchical failure recovery sequence from a FFRSM database that may correct the failure; prompting the user to select an information handling system shutdown; displaying a message to indicate that the failure is not recoverable and a new hierarchical failure recovery sequence is required.

12 . The method of claim 1 , wherein searching for the one or more recovery sequences that corresponds with the failure further comprises:
identifying the recovery sequences within local storage; and in response to not being able to identify the recovery sequences in local storage, searching for the recovery sequences that correspond to the failure from a support site, downloading a found recovery sequence at the support site, and executing the found recovery sequence at the IHS.

13 . The method of claim 1 , further comprising:
in response to a detected failure and not having access to an appropriate recovery sequence, placing the IHS into a self-healing mode; said placing the IHS within a self-healing mode further comprises:
initiating an orderly shutdown of the information handling system; and
wherein when the information handling system cannot initiate an orderly shutdown, the information handling system is forced to shut down.

14 . An information handling system comprising:
a central unit processor (CPU); at least one component that operates based on execution of a corresponding firmware component; a local storage facility that stores executable firmware utilized to operate the at least one component; and a hierarchical recovery and learning system/module comprising:
a failure detection mechanism that detects a failure of a firmware component in the information handling system; and
a hierarchical failure recovery mechanism that:
initiates a hierarchical failure recovery by executing at least one recovery sequence from a set of hierarchically ordered recovery sequences, corresponding to the detected failure;
following execution of a recovery sequence, performing, before executing a next recovery sequence, an evaluating to determine whether the detected failure was corrected by executing the recovery sequence;
in response to the failure not being corrected by executing the set of hierarchically ordered recover

Assignees

• Dell Products Lp

Inventors

• Chow Alex Chunghen
• Iyer Pushkala
• Kulkarni Yogesh Prabhakar

Classifications

• G06F11/0706 Primary

  the processing taking place on a specific hardware platform or in a specific software environment · CPC title

• G06F11/1471

  involving logging of persistent data for recovery · CPC title

• G06F11/1464

  for networked environments · CPC title

• G06F11/1469 Primary

  Backup restoration techniques · CPC title

• G06F11/0793

  Remedial or corrective actions (recovery from an exception in an instruction pipeline G06F9/3861; by retry G06F11/1402; for recovering from a failure of a protocol instance or entity H04L69/40) · CPC title