Failure recovery testing framework for microservice-based applications

Patent metadata

Field	Value
Publication number	US-2017242784-A1
Application number	US-201615048102-A
Country	US
Kind code	A1
Filing date	Feb 19, 2016
Priority date	Feb 19, 2016
Publication date	Aug 24, 2017
Grant date	—

Abstract

Systems and methods are provided for resiliency testing microservice-based applications. For example, a method for resiliency testing an application includes receiving a test script that specifies a failure scenario in a distributed microservice-based application comprising a plurality of microservices, and an asserted behavioral expectation of at least one microservice of the distributed microservice-based application in response to the specified failure scenario. The specified failure scenario is translated into fault injection rules. The fault injection rules are utilized to execute fault injection operations on messages that are exchanged between at least a first microservice and a second microservice of the distributed microservice-based application, to stage the specified failure scenario during the failure recovery testing of the distributed microservice-based application. The asserted behavioral expectation of the at least one microservice is compared with an observed behavior of the at least one microservice in response to the execution of the fault injection operations.

First claim

What is claimed is:

1 . A method for resiliency testing an application, comprising:
receiving a test script that specifies (i) a failure scenario in a distributed microservice-based application comprising a plurality of microservices; and (ii) an asserted behavioral expectation of at least one microservice of the distributed microservice-based application in response to the specified failure scenario; translating the specified failure scenario into fault injection rules; utilizing the fault injection rules to execute fault injection operations on messages that are exchanged between at least a first microservice and a second microservice of the distributed microservice-based application, to stage the specified failure scenario during the failure recovery testing of the distributed microservice-based application; and comparing the asserted behavioral expectation of the at least one microservice with an observed behavior of the at least one microservice in response to the execution of the fault injection operations.

2 . The method of claim 1 , wherein executing the fault injection rules to perform fault injection actions on messages which are exchanged between at least a first microservice and a second microservice comprises:
configuring a network proxy to identify messages that are associated with test traffic exchanged between the first microservice and the second microservice; and configuring the network proxy to execute the fault injection operations on the identified messages to stage the specified failure scenario; wherein the network proxy comprises a runtime agnostic network proxy that coordinates communication of messages between the first and second microservices.

3 . The method of claim 2 , wherein the messages that are associated with the test traffic comprise messages that are tagged with an identifier which indicates that the tagged messages are associated with the test traffic.

4 . The method of claim 1 , wherein the fault injection operations comprises at least one of: (i) a first fault injection operation that causes a message from a source microservice to a destination microservice to be dropped, and an application-level error code to be returned to the source microservice; (ii) a second fault injection operation that causes a message to be forwarded from a source microservice to a destination microservice with a specified delay time; and (iii) a third fault injection operation that causes a message from a source microservice to a destination microservice to be rewritten with specified replace bytes.

5 . The method of claim 1 , wherein comparing the asserted behavioral expectation of the at least one microservice with an observed behavior of the at least one microservice in response to the execution of the fault injection actions comprises comparing the asserted behavioral expectation of a third microservice with an observed behavior of the third microservice in response to the execution of the fault injection actions on the messages that are exchanged between the first and second microservices of the distributed microservice-based application.

6 . The method of claim 1 , wherein comparing the asserted behavioral expectation of the at least one microservice with an observed behavior of the at least one microservice in response to the execution of the fault injection operations, comprises:
persistently storing data associated with the observed behavior of the at least one microservice in an event log database; and analyzing the stored data associated with observed behavior to verify whether the observed behavior of the at least one microservice conforms to the asserted behavioral expectation of the at least one microservice.

7 . The method of claim 1 , wherein the test script is reusable for failure recovery testing of different distributed microservice-based applications.

8 . An article of manufacture comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a computer to cause the computer to perform a method for resiliency testing an application, the method comprising:
receiving a test script that specifies (i) a failure scenario in a distributed microservice-based application comprising a plurality of microservices; and (ii) an asserted behavioral expectation of at least one microservice of the distributed microservice-based application in response to the specified failure scenario; translating the specified failure scenario into fault injection rules; utilizing the fault injection rules to execute fault injection operations on messages that are exchanged between at least a first microservice and a second microservice of the distributed microservice-based application, to stage the specified failure scenario during the failure recovery testing of the distributed microservice-based application; and comparing the asserted behavioral expectation of the at least one microservice with an observed behavior of the at least one microservice in response to the execution of the fault injection operations.

9 . The article of manufacture of claim 8 , wherein the program instructions for executing the fault injection rules to perform fault injection actions on messages which are exchanged between at least a first microservice and a second microservice comprise program instructions for:
configuring a network proxy to identify messages that are associated with test traffic exchanged between the first microservice and the second microservice; and configuring the network proxy to execute the fault injection operations on the identified messages to stage the specified failure scenario; wherein the network proxy comprises a runtime agnostic network proxy that coordinates communication of messages between the first and second microservices.

10 . The article of manufacture of claim 9 , wherein the messages that are associated with the test traffic comprise messages that are tagged with an identifier which indicates that the tagged messages are associated with the test traffic.

11 . The article of manufacture of claim 8 , wherein the fault injection operations comprises at least one of: (i) a first fault injection operation that causes a message from a source microservice to a destination microservice to be dropped, and an application-level error code to be returned to the source microservice; (ii) a second fault injection operation that causes a message to be forwarded from a source microservice to a destination microservice with a specified delay time; and (iii) a third fault injection operation that causes a message from a source microservice to a destination microservice to be rewritten with specified replace bytes.

12 . The article of manufacture of claim 8 , wherein the program instructions for comparing the asserted behavioral expectation of the at least one microservice with an observed behavior of the at least one microservice in response to the execution of the fault injection actions comprise program instructions for comparing the asserted behavioral expectation of a third microservice with an observed behavior of the third microservice in response to the execution of the fault injection actions on the messages that are exchanged between the first and second microservices of the distributed microservice-based application.

13 . The article of manufacture of claim 8 , wherein the program instructions for comparing the asserted behavioral expectation of the at least one microservice with an observed behavior of the at least one microservice in response to the execution of the fault injection operations, comprise program instructions for:
persistentl

Assignees

• IBM

Inventors

• Heorhiadi Viktor
• Jamjoom Hani T
• Rajagopalan Shriram

Classifications

• G06F11/3684

  for test design, e.g. generating new test cases · CPC title

• G06F11/3692 Primary

  for test results analysis · CPC title

• G06F11/3688

  for test execution, e.g. scheduling of test suites · CPC title

• H04L43/50

  Testing arrangements · CPC title

• H04L41/0654

  using network fault recovery (ring fault isolation or reconfiguration in loop networks without recovery actions by a network management system H04L12/437) · CPC title