Persistent memory replication in RDMA-capable networks

Patent metadata

Field	Value
Publication number	US-10778767-B2
Application number	US-201715582027-A
Country	US
Kind code	B2
Filing date	Apr 28, 2017
Priority date	Apr 28, 2017
Publication date	Sep 15, 2020
Grant date	Sep 15, 2020

Abstract

A mechanism is provided in a data processing system comprising at least one processor and at least one memory, the at least one memory comprising instructions that are executed by the at least one processor and configure the at least one processor to implement a replication protocol for replicating memory from an initiator to a target. The initiator requests one or more pages of memory at the target. Responsive to the initiator receiving a page advertisement from the target, the initiator updates a list of pages at the initiator. The list of pages is replicated at the target. The initiator performs a remote direct memory access (RDMA) write to the one or more pages of memory at the target. Responsive to successful completion of the RDMA write, the initiator updates the list of pages at the initiator. Upon completion of the RDMA write the list of pages is updated at the target.

First claim

What is claimed is:

1. A method, in a data processing system comprising at least one processor and at least one memory, the at least one memory comprising instructions that are executed by the at least one processor and configure the at least one processor to implement a replication protocol for replicating memory from an initiator to a target, the method comprising:
requesting, by the initiator, one or more pages of memory at the target;
responsive to the initiator receiving a page advertisement from the target, updating a list of pages at the initiator, wherein the list of pages is replicated at the target;
performing, by the initiator, a remote direct memory access (RDMA) write to the one or more pages of memory at the target;
responsive to successful completion of the RDMA write, updating the list of pages at the initiator, wherein upon completion of the RDMA write the list of pages is updated at the target;
responsive to disconnecting from the target and subsequently reconnecting to the target, receiving a login message from the target;
sending a login reply message to the target, wherein the login reply message comprises a message code and a last initiator input/output sequence number (IOSN) that completed without error; and
responsive to receiving a login reply message from the target comprising a last target input/output initiator sequence number (IOSN) processed by target, removing all messages from a retry queue having an IOSN less than or equal to the last IOSN reported as processed by target and resuming the initiator.

2. The method of claim 1 , further comprising, responsive to successful completion of the RDMA write, sending a commit message from the initiator to the target.

3. The method of claim 1 , wherein the list of pages at the initiator is a circular queue, wherein a head index points to a next free page in an RDMA pool at the initiator, wherein a tail index points to a last free page in the RDMA pool at the initiator, and wherein a free index points to a last index in the RDMA pool that is not populated with any page.

4. The method of claim 1 , wherein performing the RDMA write comprises sending an RDMA request from the initiator to the target and placing the RDMA request in a posted queue at the initiator and wherein performing the RDMA write further comprises adding the RDMA request to a pending queue at the initiator responsive to determining the pending queue or a retry queue at the initiator is not empty, determining the target is disconnected, or determining there are not enough memory pages for the RDMA request at the target.

5. The method of claim 1 , wherein performing the RDMA write comprises sending an RDMA request from the initiator to the target and placing the RDMA request in a posted queue at the initiator and wherein performing the RDMA write further comprises adding the RDMA request to a retry queue at the initiator responsive to determining that the RDMA write failed.

6. The method of claim 1 , wherein performing the RDMA write comprises sending an RDMA request from the initiator to the target and placing the RDMA request in a posted queue at the initiator and wherein performing the RDMA write further comprises adding the RDMA request to a posted list at the initiator responsive to determining that the RDMA write succeeded.

7. The method of claim 1 , wherein performing the RDMA write comprises sending an RDMA request from the initiator to the target and placing the RDMA request in a posted queue at the initiator and wherein performing the RDMA write further comprises sending a commit message responsive to successful completion of the RDMA write.

8. The method of claim 1 , wherein the initiator and the target use input/output sequence numbers (IOSNs) to track messages.

9. The method of claim 1 , further comprising:
responsive to determining the target is blacklisted, calling an application callback with immediate failure status.

10. A computer program product comprising a non-transitory computer readable storage medium having a computer readable program stored therein, wherein the computer readable program, when executed on a computing device, causes the computing device to implement a replication protocol for replicating memory from an initiator to a target, wherein the computer readable program causes the computing device to:
request, by the initiator, one or more pages of memory at the target;
responsive to the initiator receiving a page advertisement from the target, update a list of pages at the initiator, wherein the list of pages is replicated at the target;
perform, by the initiator, a remote direct memory access (RDMA) write to the one or more pages of memory at the target;
responsive to successful completion of the RDMA write, update the list of pages at the initiator, wherein upon completion of the RDMA write the list of pages is updated at the target;
responsive to disconnecting from the target and subsequently reconnecting to the target, receive a login message from the target;
send a login reply message to the target, wherein the login reply message comprises a message code and a last initiator input/output sequence number (IOSN) that completed without error; and
responsive to receiving a login reply message from the target comprising a last target input/output initiator sequence number (IOSN) processed by target, remove all messages from a retry queue having an IOSN less than or equal to the last IOSN reported as processed by target and resume the initiator.

11. The computer program product of claim 10 , wherein the computer readable program further causes the computing device to, responsive to successful completion of the RDMA write, sending a commit message from the initiator to the target.

12. The computer program product of claim 10 , wherein the list of pages at the initiator is a circular queue, wherein a head index points to a next free page in an RDMA pool at the initiator, wherein a tail index points to a last free page in the RDMA pool at the initiator, and wherein a free index points to a last index in the RDMA pool that is not populated with any page.

13. The computer program product of claim 10 , wherein performing the RDMA write comprises sending an RDMA request from the initiator to the target and placing the RDMA request in a posted queue at the initiator and wherein performing the RDMA write further comprises adding the RDMA request to a pending queue at the initiator responsive to determining the pending queue or a retry queue at the initiator is not empty, determining the target is disconnected, or determining there are not enough memory pages for the RDMA request at the target.

14. The computer program product of claim 10 , wherein performing the RDMA write comprises sending an RDMA request from the initiator to the target and placing the RDMA request in a posted queue at the initiator and wherein performing the RDMA write further comprises adding the RDMA request to a retry queue at the initiator responsive to determining that the RDMA write failed.

15. The computer program product of claim 10 , wherein performing the RDMA write comprises sending an RDMA request from the initiator to the target and placing the RDMA request in a posted queue at the initiator and wherein performing the RDMA write further comprises adding the RDMA request to a posted list at the initiator responsive to determining that the RDMA write succeeded.

16. The computer program product of claim 10 , wherein performing the RDMA write comprises sending an RDMA request from the initiator to the target and placing the RDMA request in a posted queue at t

Assignees

• IBM

Inventors

• Chernin Leonid
• Gan-Levi Daniel
• Garmiza Ehood
• Gavrilov Constantine

Classifications

• H04L67/1097 Primary

  for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS] · CPC title

• G06F3/0613

  in relation to throughput · CPC title

• G06F13/28

  using burst mode transfer, e.g. direct memory access {DMA}, cycle steal (G06F13/32 takes precedence) · CPC title

• H04L67/1095

  Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes · CPC title

• G06F3/0611

  in relation to response time · CPC title