Error handling for services requiring guaranteed ordering of asynchronous operations in a distributed environment

Patent metadata

Field	Value
Publication number	US-2019205221-A1
Application number	US-201715857596-A
Country	US
Kind code	A1
Filing date	Dec 28, 2017
Priority date	Dec 28, 2017
Publication date	Jul 4, 2019
Grant date	—

Abstract

When a particular event record requires a particular service, but that service has failed for the event, an error record that includes an aggregate identifier (AgID) is stored for the event in an error queue. Storing an error record in the error queue causes the service to notify a remediation team to fix the failed event. All events with the AgID in the error record are put on hold and an error record is created for each of these events. The remediation team generates a fixed version of the event record and causes the system to retry the failed service for the event based on the fixed version. If the fixed version of the event is successfully processed, then any other events with the same AgID that have error records in the error queue, are routed in order of enqueue time through the event processor to be handled.

First claim

What is claimed is:

1 . A method comprising:
receiving a first request to perform a first multi-system operation that involves:
a first task to be performed by a first service, and
a second task to be performed by a second service asynchronously relative to performance of the first task;
wherein the first task involves an item identified by a first aggregate identifier; after the first service has completed performance of the first task, receiving a first event record associated with the first multi-system operation, wherein the first event record includes the first aggregate identifier; in response to receiving the first event record, determining whether the first aggregate identifier is associated with any error queue record in an error queue; responsive to determining that the first aggregate identifier is associated with a first error queue record in the error queue, adding a second error queue record for the first event record to the error queue without causing the second service to perform the second task; receiving a second request to perform a second multi-system operation that involves:
a third task to be performed by the first service, and
a fourth task to be performed by the second service asynchronously relative to performance of the third task;
wherein the third task involves a second item identified by a second aggregate identifier; after the first service has completed performance of the third task, receiving a second event record associated with the second multi-system operation, wherein the second event record includes the second aggregate identifier; after the second error queue record has been added to the error queue, and in response to receiving the second event record, determining whether the second aggregate identifier is associated with any error queue record in the error queue; responsive to determining that the second aggregate identifier is not associated with any error queue record in the error queue, causing the second service to perform the fourth task; wherein the method is performed by one or more computing devices.

2 . The method of claim 1 , further comprising, prior to receiving the first request to perform a first multi-system operation:
receiving a third request to perform a third multi-system operation that involves:
a fifth task to be performed by the first service, and
a sixth task to be performed by the second service asynchronously relative to performance of the fifth task;
wherein the fifth task involves the item identified by the first aggregate identifier; after the first service has completed performance of the fifth task, receiving a third event record associated with the third multi-system operation, wherein the third event record includes the first aggregate identifier; in response to receiving the third event record, determining whether the first aggregate identifier is associated with any error queue record in the error queue; responsive to determining that the first aggregate identifier is not associated with any error queue record in the error queue, causing the second service to initiate the sixth task; receiving a failure notification from the second service indicating a failure to complete the sixth task; and responsive to receiving the failure notification from the second service indicating the failure to complete the sixth task, adding the first error queue record to the error queue; wherein the first error queue record is associated with the first aggregate identifier based on the third event record including the first aggregate identifier.

3 . The method of claim 2 , further comprising, after adding the second error queue record for the first event record to the error queue:
receiving a request to re-initiate the sixth task; responsive to receiving the request to re-initiate the sixth task, causing the second service to perform the sixth task.

4 . The method of claim 3 , further comprising, after causing the second service to perform the sixth task:
detecting that the sixth task was completed successfully; responsive to detecting that the sixth task was completed successfully:
determining which error queue record, of a set of one or more records in the error queue having the first aggregate identifier, is associated with an earliest enqueue timestamp;
wherein each error queue record includes at least: an enqueue timestamp, an aggregate identifier, and information identifying a particular task represented by the error queue record;
responsive to determining that the second error queue record is associated with the earliest enqueue timestamp among the set of one or more records, causing the second service to perform the second task based on the second error queue record including information identifying the second task.

5 . The method of claim 2 , wherein the failure to complete the sixth task was caused, at least in part, by at least one of the first or second services.

6 . The method of claim 2 , wherein the failure to complete the sixth task was caused, at least in part, by a service other than the first or second services.

7 . The method of claim 1 , wherein receiving the first event record is performed by the second service.

8 . The method of claim 1 , wherein receiving the first event record is performed by a service other than the second service.

9 . The method of claim 1 , wherein the second service for all event records associated with a particular aggregate identifier are performed by a particular processing thread.

10 . One or more non-transitory computer-readable media storing instructions which, when executed by one or more processors, cause:
receiving a first request to perform a first multi-system operation that involves:
a first task to be performed by a first service, and
a second task to be performed by a second service asynchronously relative to performance of the first task;
wherein the first task involves an item identified by a first aggregate identifier; after the first service has completed performance of the first task, receiving a first event record associated with the first multi-system operation, wherein the first event record includes the first aggregate identifier; in response to receiving the first event record, determining whether the first aggregate identifier is associated with any error queue record in an error queue; responsive to determining that the first aggregate identifier is associated with a first error queue record in the error queue, adding a second error queue record for the first event record to the error queue without causing the second service to perform the second task; receiving a second request to perform a second multi-system operation that involves:
a third task to be performed by the first service, and
a fourth task to be performed by the second service asynchronously relative to performance of the third task;
wherein the third task involves a second item identified by a second aggregate identifier; after the first service has completed performance of the third task, receiving a second event record associated with the second multi-system operation, wherein the second event record includes the second aggregate identifier; after the second error queue record has been added to the error queue, and in response to receiving the second event record, determining whether the second aggregate identifier is associated with any error queue record in the error queue; responsive to determining that the second aggregate identifier is not associated with any error queue record in the error queue, causing the second service to perform the fourth task.

11 . The one or more non-transitory computer-readable media of

Assignees

• Lendingclub Corp

Inventors

• Taubler David
• Han Hyunsuk
• Nikitina Yana
• Daum John
• Pradjinata Wira
• Petrunya Igor

Classifications

• G06F2201/835

  Timestamp · CPC title

• G06F11/1471 Primary

  involving logging of persistent data for recovery · CPC title

• G06F16/273 Primary

  Asynchronous replication or reconciliation · CPC title

• G06F16/25

  Integrating or interfacing systems involving database management systems · CPC title

• G06F9/5038

  considering the execution order of a plurality of tasks, e.g. taking priority or time dependency constraints into consideration (scheduling strategies G06F9/4881 and subgroups) · CPC title