Data exchange and processing synchronization in distributed systems

Patent metadata

Field	Value
Publication number	US-11689618-B2
Application number	US-201916718685-A
Country	US
Kind code	B2
Filing date	Dec 18, 2019
Priority date	Dec 18, 2019
Publication date	Jun 27, 2023
Grant date	Jun 27, 2023

Abstract

Systems, methods, techniques and apparatuses of asynchronous communication is distributed systems are disclosed. One exemplary embodiment is a method determining, with a plurality of agent nodes structured to communicate asynchronously in a distributed system, a first set of iterations including an iteration determined by each of the plurality of agent nodes; determining, with a first agent node of the plurality of agent nodes, a local vector clock; receiving, with the first agent node, a first iteration of the first set of iterations and a remote vector clock determined based on the first iteration; updating, with the first agent node, the local vector clock based on the received remote vector clock; and determining a first iteration of a second set of iterations based on the first set of iterations after determining all iterations of the first set of iterations have been received based on the local vector clock.

First claim

What is claimed is:

1. A method comprising:
determining, with a plurality of agent nodes structured to communicate asynchronously in a distributed system, a first set of iterations including an iteration determined by each of the plurality of agent nodes, the plurality of agent nodes being a microgrid control system, a microgrid controller, or an industrial automation controller;
determining, with a first agent node of the plurality of agent nodes, a local vector clock;
receiving, with the first agent node, a first iteration of the first set of iterations and a remote vector clock determined based on the first iteration;
updating, with the first agent node, the local vector clock based on the received remote vector clock; and
determining, with the first agent node, a first iteration of a second set of iterations based on the first set of iterations after determining all iterations of the first set of iterations have been received based on the local vector clock;
wherein the local vector clock includes a plurality of elements, wherein a first element of the local vector clock corresponds to a number of iterations determined with the first agent node, and wherein each of the remaining elements of the local vector clock corresponds to a number of received iterations determined by one of the other agent nodes.

2. The method of claim 1 , further comprising transmitting, with the first agent node, the first iteration of the first set of iterations and the updated local vector clock in response to determining all iterations of the first set of iterations determined by the other agents nodes have not been received.

3. The method of claim 1 , wherein determining, with the first agent node, the first iteration of the second set of iterations occurs in response to the first agent node determining the first iteration is not a final iteration, and wherein the method comprises updating the local vector clock based on the first iteration of the second set of iterations, and transmitting, with the first agent node, the first iteration of the second set of iterations and the local vector clock updated based on the first iteration of the second set of iterations.

4. The method of claim 1 , wherein the first iteration of the second set of iterations is determined by the first agent node before at least one agent node of the plurality of agent nodes has received all iterations of the first set of iterations.

5. The method of claim 1 , wherein the plurality of agent nodes is structured to communicate without being synchronized to a common time source.

6. The method of claim 1 , further comprising repeatedly receiving iterations, updating the local vector clock, and determining a new iteration based on the local vector clock until the first agent node determines the most recently determined iteration is a final iteration.

7. The method of claim 6 , wherein the plurality of agent nodes are microgrid controllers, wherein the final iteration is a state estimation, and wherein the method comprises operating, with the first agent node, a controllable device of a microgrid based on the final iteration.

8. The method of claim 6 , wherein the plurality of agent nodes are industrial automation controllers, wherein the final iteration is a state estimation, and wherein the method comprises operating, with the first agent node, a controllable device of an industrial automation system based on the final iteration.

9. A distributed system comprising:
a plurality of agent nodes including a first agent node, the plurality of agent nodes being a microgrid control system, a microgrid controller, or an industrial automation controller; and
a communication network structured to allow the plurality of agent nodes to communicate asynchronously;
wherein the plurality of agent nodes each include a set of instructions stored on a memory device which, when executed by a processing device of the corresponding agent node, is effective to determine a first set of iterations including an iteration determined by each of the plurality of agent nodes, wherein the first agent node is structured to execute the set of instructions stored on the memory device of the first agent node with the processing device of the first agent node effective to:
determine a local vector clock;
receive a first iteration of the first set of iterations and a remote vector clock determined based on the first iteration;
update the local vector clock based on the received remote vector clock; and
determine a first iteration of a second set of iterations based on the first set of iterations after determining all iterations of the first set of iterations have been received based on the local vector clock;

wherein the local vector clock includes a plurality of elements, wherein a first element of the local vector clock corresponds to a number of iterations determined with the first agent node, and wherein each of the remaining elements of the local vector clock corresponds to a number of received iterations determined by one of the other agent nodes.

10. A method, comprising:
determining, with a first agent node, a first iteration in a first set of iterations, the first agent node being the first agent node in a plurality of agent nodes of a distributed system communicating asynchronously, the first set of iterations comprising iterations determined by each of the plurality of agent nodes and the plurality of agent nodes being a microgrid control system, a microgrid controller, or an industrial automation controller;
determining, with the first agent node, a local vector clock;
receiving, with the first agent node, a respective iteration of the first set of iterations;
receiving, with the first agent node, a remote vector clock determined based on the respective iteration;
updating, with the first agent node, the local vector clock based on the received remote vector clock; and
determining, with the first agent node, a first iteration of a second set of iterations based on the first set of iterations, the determining being after determining that all iterations of the first set of iterations have been received based on the local vector clock:

wherein the local vector clock includes a plurality of elements, wherein a first element of the local vector clock corresponds to a number of iterations determined with the first agent node, and wherein each of the remaining elements of the local vector clock corresponds to a number of received iterations determined by one of the other agent nodes.

11. The method of claim 10 , the method further comprising transmitting, with the first agent node, the first iteration of the first set of iterations and the updated local vector clock in response to determining all iterations of the first set of iterations determined by the other agents nodes have not been received.

12. The method of claim 10 , wherein determining, with the first agent node, the first iteration of the second set of iterations occurs in response to the first agent node determining the first iteration is not a final iteration, and wherein the method comprises updating the local vector clock based on the first iteration of the second set of iterations, and transmitting, with the first agent node, the first iteration of the second set of iterations and the local vector clock updated based on the first iteration of the second set of iterations.

13. A distributed system, comprising:
a plurality of agent nodes configured to communicate asynchronously in the distributed system, the plurality of agent nodes being a microgrid control system, a microgrid controller, or an industrial automation controller, the plurality of agent nodes c

Assignees

• Hitachi Energy Switzerland Ag

Inventors

• Cintuglu Mehmet H
• Ishchenko Dmitry

Classifications

• H04L67/1095 Primary

  Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes · CPC title

• G06F17/16

  Matrix or vector computation {, e.g. matrix-matrix or matrix-vector multiplication, matrix factorization (matrix transposition G06F7/78)} · CPC title

• H04L67/10 Primary

  in which an application is distributed across nodes in the network (software deployment G06F8/60; multiprogramming arrangements G06F9/46) · CPC title

• H04L67/1074

  for supporting data block transmission mechanisms (file transfer H04L67/06) · CPC title

• H04L67/12

  specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks · CPC title