Payload direct memory storing (PDMS) for remote direct memory access (RDMA)

Patent metadata

Field	Value
Publication number	US-12407766-B2
Application number	US-202217902150-A
Country	US
Kind code	B2
Filing date	Sep 2, 2022
Priority date	Sep 2, 2022
Publication date	Sep 2, 2025
Grant date	Sep 2, 2025

Abstract

Technologies for payload direct memory storing (PDMS) for out-of-order delivery of packets in remote direct memory access (RDMA) are described. A responder device includes an RDMA transport layer that can receive packets out of order and allow direct data placement of packet data in order. The responder device receives a first packet with a first packet number and first location information. The responder device stores first packet data to a first location according to the first location information. The responder device also receives a second packet and stores second packet data to a second location according to the second location information. A second packet number indicates that the first packet is received out of order. The first and second packet data are stored in order. The responder device can provide an indication that a message has arrived in response to determining that all packets of the message have arrived.

First claim

What is claimed is:

1. A method of operating a first device, the method comprising:
receiving a first packet of a message in a single remote direct memory access (RDMA) transport stream from a second device over a first route between the first device and the second device, the first packet comprising a first packet number and first location information;
storing first packet data of the first packet at a first location according to the first location information;
receiving a second packet of the message over a second route between the first device and the second device, the second packet comprising a second packet number and second location information, wherein the first route and the second route are different, wherein the second packet number indicates that the first packet is received out of order relative to the second packet; and
storing second packet data of the second packet at a second location according to the second location information, wherein the first location and the second location store the first packet data and the second packet data in order.

2. The method of claim 1 , wherein the first packet number is a higher packet sequential number (PSN) than the second packet number.

3. The method of claim 1 , further comprising providing an indication that the message has arrived in response to determining that all packets of the message have arrived, wherein the message is at least one of an RDMA send request, an RDMA write request, an RDMA read request, or an RDMA atomic request.

4. The method of claim 1 , wherein the first location is a first memory location in a memory, wherein the second location is a second memory location in the memory.

5. The method of claim 1 , wherein the first location and the second location are locations in a data buffer.

6. The method of claim 1 , wherein the message is an RDMA send request, wherein the first packet comprises a header field with the first location information, wherein the first location information identifies an offset of the first packet within the message.

7. The method of claim 1 , further comprising:
determining that all packets of the message have arrived; and
providing an indication that the message has arrived in response to determining that all packets of the message have arrived.

8. The method of claim 7 , further comprising:
maintaining a list of packets of the message that have successfully arrived.

9. The method of claim 1 , further comprising:
receiving a third packet of the message from the second device, the third packet comprising a first packet sequential number (PSN) of the message, wherein the first PSN is to identify a message context of the message, wherein the third packet is received before the first packet and the second packet.

10. The method of claim 9 , further comprising:
selecting a work queue element (WQE) in a data buffer for the message in response to receiving the third packet, wherein the message is for an RDMA send request, wherein the first location information identifies a first offset of the first packet data within the message, and wherein the second location information identifies a second offset of the second packet data within the message;
scattering the first packet data to the data buffer according to the first offset;
scattering the second packet data to the data buffer according to the second offset; and
updating a completion status of a completion queue entry (CQE) associated with the WQE.

11. A method of operating a first device, the method comprising:
receiving a first packet of a message in a single remote direct memory access (RDMA) transport stream from a second device, the first packet comprising a first packet number and first location information;
storing first packet data of the first packet at a first location according to the first location information;
receiving a second packet of the message, the second packet comprising a second packet number and second location information, wherein the second packet number indicates that the first packet is received out of order relative to the second packet;
storing second packet data of the second packet at a second location according to the second location information, wherein the first location and the second location store the first packet data and the second packet data in order; and
receiving a third packet of the message from the second device, the third packet comprising a first message identifier to identify a message context of the message, wherein the third packet is received before the first packet and the second packet.

12. A first device comprising:
a memory or data buffer to store packet data;
a processing device coupled to the memory or data buffer, the processing device to:
receive and accept packets of a message in a single remote direct memory access (RDMA) transport stream regardless of an order of receipt of the packets; and
store packet data of the packets in order in the memory or data buffer using offset information included in the packets, wherein the processing device comprises a transport layer, wherein the transport layer is to provide an indication that the message has arrived to a higher layer than the transport layer, in response to determining that all packets of the message have arrived.

13. The first device of claim 12 , wherein the first device is a network adapter.

14. The first device of claim 12 , wherein the first device is a network interface card (NIC).

15. A first device comprising:
a memory or data buffer to store packet data;
a processing device coupled to the memory or data buffer, the processing device to:
receive and accept packets of a message in a single remote direct memory access (RDMA) transport stream regardless of an order of receipt of the packets;
store packet data of the packets in order in the memory or data buffer using offset information included in the packets;
receive a first packet with a first packet sequential number (PSN) and a first offset within the message;
receive a second packet with a second PSN and a second offset within the message, wherein the first PSN is higher than the second PSN; and
store first packet data of the first packet and second packet data of the second packet in the memory or data buffer in order using the first offset and the second offset, respectively.

16. The first device of claim 15 , wherein the first packet is received from a second device over a first route between the first device and the second device, wherein the second packet is received from the second device over a second route between the first device and the second device, wherein the first route and the second route are different.

17. The first device of claim 12 , wherein the processing device is to:
receive a first packet with a message number (MSN) and a first offset within the message;
receive a second packet with the MSN and a second offset within the message; and
store first packet data of the first packet and second packet data of the second packet in the memory or data buffer in order using the first offset and the second offset, respectively.

18. A communication system comprising:
a requestor device;
a responder device; and
responder memory coupled to the responder device, wherein the responder device is to:
receive a first packet of a message in a single remote direct memory access (RDMA) transport stream from the requestor device over a first route between the requestor device and the responder device, the first packet comprising a first packet number and first location information;
store firs

Assignees

• Mellanox Technologies Ltd

Inventors

• Friedman Yamin
• Shahar Ariel
• Borshteen Idan
• Moyal Roee

Classifications

• G06F13/28 Primary

  using burst mode transfer, e.g. direct memory access {DMA}, cycle steal (G06F13/32 takes precedence) · CPC title

• G06F15/167

  using a common memory, e.g. mailbox · CPC title

• H04L49/90

  Buffering arrangements · CPC title

• G06F15/17331

  Distributed shared memory [DSM], e.g. remote direct memory access [RDMA] · CPC title

• H04L69/22 Primary

  Parsing or analysis of headers · CPC title