Fault isolation and identification in versioned microservices

Patent metadata

Field	Value
Publication number	US-10275329-B2
Application number	US-201715428525-A
Country	US
Kind code	B2
Filing date	Feb 9, 2017
Priority date	Feb 9, 2017
Publication date	Apr 30, 2019
Grant date	Apr 30, 2019

Abstract

A method for identifying and isolating faults in versioned microservices includes a request replicator receiving an original request, and determining whether to replicate the original request. The request replicator replicates the original request creating one or more replicated requests, including a first replicated request. In an example, the request replicator dispatches the original request to a stable production system, and dispatches the first replicated request to a first modified production system. The stable production system produces a first reply to the original request. The first modified production system produces a second reply to the first replicated request. A fault detector performs a comparison of the second reply and the first reply and determines, based on the comparison, that the first modified production system has a verification status. Then, the stable production system is replaced with first modified production system.

First claim

The invention is claimed as follows:

1. A microservice fault isolation method comprising:
receiving, at a request replicator, an original request;
determining, by the request replicator, whether to replicate the original request;
replicating, by the request replicator, the original request, creating one or more replicated requests, including a first replicated request;
dispatching the original request to a stable production system, wherein the stable production system includes a first plurality of microservices running on a validated set of container images;
dispatching the first replicated request to a first modified production system, wherein the first modified production system includes a second plurality of microservices running on an unvalidated set of container images;
producing, by the stable production system, a first reply to the original request;
producing, by the first modified production system, a second reply to the first replicated request;
performing, by a fault detector, a comparison of the second reply and the first reply;
determining, by the fault detector, that based on the comparison the first modified production system has a verification status; and
replacing the stable production system with the first modified production system.

2. The microservice fault isolation method of claim 1 , wherein the verification status indicates that the second reply is identical to the first reply.

3. The microservice fault isolation method of claim 1 , wherein an allowed threshold of accuracy is determined by an administrator, and
wherein the verification status indicates whether the second reply is within the allowed threshold of accuracy from the first reply.

4. The microservice fault isolation method of claim 1 , wherein an allowed threshold of time is determined by an administrator, and
wherein the verification status indicates whether the second reply is received within the allowed threshold of time after the first reply.

5. The microservice fault isolation method of claim 1 , further comprising:
dispatching a second replicated request to a second modified production system, wherein the second modified production system runs on a complete unvalidated set of container images; and
dispatching a third replicated request to a third modified production system, wherein the third modified production system runs on a partial unvalidated set of container images.

6. The microservice fault isolation method of claim 5 , further comprising:
producing, by the second modified production system, a third reply to the second replicated request; and
producing, by the third modified production system, a fourth reply to the third replicated request.

7. The microservice fault isolation method of claim 6 , further comprising:
performing, by the fault detector, a comparison of the third reply to the first reply; and
performing, by the fault detector, a comparison of the fourth reply to the first reply.

8. The microservice fault isolation method of claim 1 , further comprising:
sending, by a client device, the original request to the request replicator.

9. The microservice fault isolation method of claim 8 , further comprising:
communicating, by the stable production system, the first reply back to the client device in response to the original request.

10. The microservice fault isolation method of claim 1 , wherein a first quantity of modified production systems is determined by a second quantity of unvalidated container images that are identified as requiring verification.

11. The microservice fault isolation method of claim 1 , further comprising:
reporting, by the fault detector, the verification status to an administrator.

12. The microservice fault isolation method of claim 1 , further comprising:
processing, by the stable production system and the first modified production system, the original request and the first replicated request, respectively, in real time.

13. The microservice fault isolation method of claim 1 , wherein the original request is a web-browser request.

14. The microservice fault isolation method of claim 1 , wherein the original request is a web-application request.

15. The microservice fault isolation method of claim 1 , further comprising:
determining, by the request replicator, to replicate the original request based on privacy information in the original request.

16. The microservice fault isolation method of claim 1 , further comprising:
selecting, by the request replicator, which original requests to replicate.

17. The microservice fault isolation method of claim 1 , further comprising:
determining, by the request replicator, to replicate the original request based on privacy information in the original request; and
selecting, by the request replicator, which original requests to replicate.

18. A system comprising:
one or more processors;
a stable production system that includes a first plurality of microservices running on a validated set of container images;
a first modified production system that includes a second plurality of microservices running on an unvalidated set of container images;
a request replicator; and
a fault detector,
wherein the one or more processors, when executed:
receive, at the request replicator, an original request,
determine, by the request replicator, whether to replicate the original request,
replicate, by the request replicator, the original request, creating one or more replicated requests, including a first replicated request,
dispatch the original request to the stable production system,
dispatch the first replicated request to the first modified production system,
produce, by the stable production system, a first reply to the original request,
produce, by the first modified production system, a second reply to the first replicated request,
perform, by the fault detector, a comparison of the second reply and the first reply,
determine, by the fault detector, that based on the comparison the first modified production system has a verification status, and
replace the stable production system with the first modified production system.

19. The system of claim 18 , further comprising:
a second modified production system; and
a third modified production system.

20. A non-transitory machine readable medium storing instructions, which when executed by one or more physical processors, cause the one or more physical processors to:
receive, at a request replicator, an original request;
determine, by the request replicator, whether to replicate the original request;
replicate, by the request replicator, the original request, creating one or more replicated requests, including a first replicated request;
dispatch the original request to a stable production system;
dispatch the first replicated request to a first modified production system;
produce, by the stable production system, a first reply to the original request;
produce, by the first modified production system, a second reply to the first replicated request;
perform, by a fault detector, a comparison of the second reply and the first reply;
determine, by the fault detector, that based on the comparison the first modified production system has a verification status; and
replace the stable production system with the first modified production system.

Assignees

• Red Hat Inc

Inventors

• Chen Huamin
• Vyas Jay

Classifications

• G06F11/2038 Primary

  with a single idle spare processing component · CPC title

• G06F11/1487

  using N-version programming · CPC title

• G06F11/1629

  Error detection by comparing the output of redundant processing systems · CPC title

• G06F11/14 Primary

  Error detection or correction of the data by redundancy in operations (error detection or correction of the data by redundancy in hardware G06F11/16) · CPC title

• G06F11/3688

  for test execution, e.g. scheduling of test suites · CPC title