Network data transactions using posted and non-posted operations

Patent metadata

Field	Value
Publication number	US-10552367-B2
Application number	US-201715659876-A
Country	US
Kind code	B2
Filing date	Jul 26, 2017
Priority date	Jul 26, 2017
Publication date	Feb 4, 2020
Grant date	Feb 4, 2020

Abstract

Communication apparatus includes a host interface, configured to be coupled to a host processor having a host memory, and a network interface, which is configured to receive over a network from a sending node data packets conveying operations for execution in a sequential order on a predefined queue pair (QP), including at least a first packet conveying a posted write operation and a second packet conveying a non-posted write operation. Packet processing circuitry is configured to execute the posted write operation in accordance with the sequential order so as to write first data to the host memory prior to the execution of any subsequent operations in the sequential order, and to execute the non-posted write operation so as to write second data to the host memory while allowing one or more of the subsequent operations in the sequential order to be executed prior to completion of writing the second data.

First claim

The invention claimed is:

1. Communication apparatus, comprising:
a host interface, configured to be coupled to a host processor having a host memory;
a network interface, which is configured to receive over a network from a sending node data packets conveying operations for execution in a sequential order on a predefined queue pair (QP), the packets comprising respective transport headers, which contain respective operation codes identifying different types of the operations, including at least a first packet conveying a first write operation of a first type identified by a first operation code and a second packet conveying a second write operation of a second type identified a second operation code; and
packet processing circuitry, which is configured to parse the transport headers and, responsively to the first and second operation codes in the respective transport headers, to execute the first write operation in accordance with the sequential order so as to write first data to the host memory prior to the execution of any subsequent operations in the sequential order, and to execute the write operation so as to write second data to the host memory while allowing one or more of the subsequent operations in the sequential order to be executed prior to completion of writing the second data.

2. The apparatus according to claim 1 , wherein the second write operation comprises a remote direct memory access (RDMA) write operation.

3. The apparatus according to claim 2 , wherein the RDMA write operation comprises a non-posted RDMA write with immediate operation, which causes the packet processing circuitry to write the second data to the host memory while queuing the RDMA write in a response queue, and to write a completion queue element (CQE) to the host memory only after execution and completion of one or more preceding operations of the second type.

4. The apparatus according to claim 1 , wherein the second write operation comprises a non-posted atomic operation, which causes the packet processing circuitry to read a first value from a specified address in the host memory and responsively to the first value, to write a second value to the specified address.

5. The apparatus according to claim 4 , wherein the packet processing circuitry is configured to queue operations of the second type for execution in a response queue, including the non-posted atomic operation, and to block execution of the non-posted atomic operation and acknowledgment of the non-posted atomic operation to the sending node until at least one preceding operation of the second type in the response queue has been completed.

6. The apparatus according to claim 1 , wherein the second write operation comprises a non-posted send operation, which causes the packet processing circuitry to read a work queue element (WQE) posted by the host processor in a receive queue, to write the second data to a buffer specified by the WQE, and after writing the second data, to write a completion queue element (CQE) to the host memory.

7. The apparatus according to claim 6 , wherein the packet processing circuitry is configured to queue operations of the second type for execution in a response queue, including the non-posted send operation, and to acknowledge the non-posted send operation to the sending node only after any preceding operations in the response queue have been completed.

8. The apparatus according to claim 1 , wherein the packet processing circuitry is configured to queue operations of the second type for execution in a response queue, including the second write operation, and to block execution of the second write operation and acknowledgment of the second write operation to the sending node until at least one preceding operation has been completed.

9. The apparatus according to claim 8 , wherein the at least one preceding operation comprises a flush operation.

10. The apparatus according to claim 8 , wherein the at least one preceding operation comprises an RDMA read operation.

11. The apparatus according to claim 8 , wherein the at least one preceding operation comprises a write operation of the second type.

12. The apparatus according to claim 1 , wherein the packet processing circuitry is configured to queue operations of the second type for execution in a response queue, including the second write operation, and to write a completion queue element (CQE) to the host memory upon writing the second data to the host memory, while enabling the host processor to access the CQE only after completion of any preceding operations in the response queue.

13. The apparatus according to claim 12 , wherein the packet processing circuitry is configured to queue the CQE in a completion queue in order with CQEs of the preceding operations in the response queue.

14. The apparatus according to claim 1 , wherein the host memory comprises a volatile memory and a target memory, and
wherein the data packets include a third packet conveying a flush operation following the first write operation and preceding the second write operation in the sequential order, wherein the flush operation causes the first data to be flushed from the volatile memory to the target memory, and
wherein the packet processing circuitry is configured to delay the execution of the second write operation until a notification that the flush operation has been completed is received via the host interface.

15. The apparatus according to claim 14 , wherein the packet processing circuitry is configured to queue the flush operation in a response queue, and to transmit a flush acknowledgment of the flush operation to the sending node only after having received the notification that the flush operation was completed, and to queue the second write operation in the response queue after the flush operation, so that the second write operation is executed and acknowledged to the sending node only after the flush acknowledgment has been transmitted.

16. The apparatus according to claim 14 , wherein the first data comprise an update to a record in a database maintained in the host memory, and wherein the second data comprise an update to a pointer, which points to the updated record.

17. The apparatus according to claim 16 , wherein the data packets include a fourth packet conveying a further flush operation following the second write operation in the sequential order, wherein the further flush operation causes the update to the pointer to be flushed from the volatile memory to the target memory, and
wherein the packet processing circuitry is configured to receive an additional notification via the host interface that the further flush operation has been completed, and to transmit a flush acknowledgment of the further flush operation to the sending node after receiving the additional notification so as to inform the sending node that a two-phase commit of the update has been completed.

18. The apparatus according to claim 17 , wherein the data packets include a fifth packet conveying a send operation of the second type following the further flush operation in the sequential order, and
wherein the packet processing circuitry is configured, after executing the send operation, to write a completion queue element (CQE) to the host memory so as to inform the host processor that the two-phase commit of the update has been completed.

19. The apparatus according to claim 1 , wherein the packet processing circuitry is configured to negotiate with the sending node so as to set a maximum buffer allocation for outstanding write operations of the second type,

Assignees

• Mellanox Technologies Ltd

Inventors

• Burstein Idan
• Crupnicoff Diego

Classifications

• G06F15/17331

  Distributed shared memory [DSM], e.g. remote direct memory access [RDMA] · CPC title

• H04L67/42

  Electricity · mapped topic

• G06F15/167 Primary

  using a common memory, e.g. mailbox · CPC title

• H04L67/1097 Primary

  for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS] · CPC title

• H04L67/01

  Protocols · CPC title