Injecting faults at select execution points of distributed applications

Patent metadata

Field	Value
Publication number	US-9483383-B2
Application number	US-201314097713-A
Country	US
Kind code	B2
Filing date	Dec 5, 2013
Priority date	Dec 5, 2013
Publication date	Nov 1, 2016
Grant date	Nov 1, 2016

Abstract

Methods, systems, and articles of manufacture for injecting faults at select execution points of distributed applications are provided herein. A method includes monitoring a run-time state of each of multiple components of a distributed application to determine one or more sequence of events that triggers a fault injection point at one of the multiple components; defining a fault injection scenario in a specification based on said monitoring, wherein said fault injection scenario comprises a description of one or more sequence of events during which an intended fault is to be injected to a target component of the multiple components at one selected event; and executing the fault injection defined in the specification to perform injection of the intended fault during run-time of the distributed application.

First claim

What is claimed is:

1. A method comprising:
monitoring a run-time state of each of multiple distributed components of a distributed application that comprises a variable number of multiple states, based on collected execution traces of the distributed application, to determine one or more sequence of events that triggers a fault injection point at each of the multiple components, wherein said one or more sequence of events comprises a sequence of log events;
defining a scenario of multiple fault injections in a specification based on said monitoring, wherein said scenario comprises a description of one or more sequence of events during which each of the multiple faults is to be injected across the multiple distributed components at one selected event, and wherein said defining comprises defining the scenario based on one or more event dependencies discovered from said monitoring; and
executing the multiple fault injections defined in the specification concurrently across the multiple distributed components of the distributed application during run-time of the distributed application;
wherein said monitoring, said defining, and said executing are carried out by at least one computing device.

2. The method of claim 1 , wherein said monitoring comprises monitoring the run-time state of each of the multiple distributed components of the distributed application under multiple workloads.

3. The method of claim 1 , wherein each of the multiple faults comprises at least one of an application crash, network congestion, a lack of memory and/or disk space, a memory, disk or network corruption and a configuration error.

4. The method of claim 1 , wherein each of the multiple distributed components comprises at least one of a given function, a given process, a virtual machine, and a physical machine.

5. The method of claim 1 , comprising:
analyzing results from said multiple fault injections.

6. The method of claim 5 , wherein said analyzing comprises collecting multiple logs derived from said multiple fault injections.

7. The method of claim 6 , comprising:
displaying said multiple logs to a user.

8. The method of claim 5 , wherein said analyzing comprises generating statistics pertaining to said multiple fault injections.

9. An article of manufacture comprising a non-transitory computer readable storage medium having computer readable instructions tangibly embodied thereon which, when implemented, cause a computer to carry out a plurality of method steps comprising:
monitoring a run-time state of each of multiple distributed components of a distributed application that comprises a variable number of multiple states, based on collected execution traces of the distributed application, to determine one or more sequence of events that triggers a fault injection point at each of the multiple components, wherein said one or more sequence of events comprises a sequence of log events;
defining a scenario of multiple fault injections in a specification based on said monitoring, wherein said scenario comprises a description of one or more sequence of events during which each of the multiple faults is to be injected across the multiple distributed components at one selected event, and wherein said defining comprises defining the scenario based on one or more event dependencies discovered from said monitoring; and
executing the multiple fault injections defined in the specification concurrently across the multiple distributed components of the distributed application during run-time of the distributed application.

10. The article of manufacture of claim 9 , wherein said monitoring comprises monitoring the run-time state of each of the multiple distributed components of the distributed application under multiple workloads.

11. A system comprising:
a memory; and
at least one processor coupled to the memory and configured for:
monitoring a run-time state of each of multiple distributed components of a distributed application that comprises a variable number of multiple states, based on collected execution traces of the distributed application, to determine one or more sequence of events that triggers a fault injection point at each of the multiple components, wherein said one or more sequence of events comprises a sequence of log events;
defining a scenario of multiple fault injections in a specification based on said monitoring, wherein said scenario comprises a description of one or more sequence of events during which each of the multiple faults is to be injected across the multiple distributed components at one selected event, and wherein said defining comprises defining the scenario based on one or more event dependencies discovered from said monitoring; and
executing the multiple fault injections defined in the specification concurrently across the multiple distributed components of the distributed application during run-time of the distributed application.

Assignees

• IBM

Inventors

• Baset Salman A
• Pham Cuong M
• Ramasamy Harigovind V
• Singh Manas
• Tak Byung Chul
• Tang Chunqiang
• Wang Long

Classifications

• G06F11/3644 Primary

  by instrumenting at runtime · CPC title

• G06F11/2033

  switching over of hardware resources · CPC title

• G06F11/3006

  where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems (multiprogramming arrangements G06F9/46; allocation of resources G06F9/50) · CPC title

• H04L43/0805

  by checking availability · CPC title

• G06F11/3612

  by runtime analysis (performance monitoring G06F11/3466) · CPC title