Reliable out-of order end-to-end protocol with robust window state overflow management and a multi-node system using same

Patent metadata

Field	Value
Publication number	US-10044626-B2
Application number	US-201514757993-A
Country	US
Kind code	B2
Filing date	Dec 24, 2015
Priority date	Dec 24, 2015
Publication date	Aug 7, 2018
Grant date	Aug 7, 2018

Abstract

In an embodiment, an out-of-order, reliable, end-to-end protocol is provided that can enable direct user-level data placement and atomic operations between nodes of a multi-node network. The protocol may be optimized for low-loss environments such as High Performance Computing (HPC) applications, and may enable loss detection and de-duplication of packets through the use of a robust window state manager at a target node. A multi-node network implementing the protocol may have increased system reliability, packet throughput, and increased tolerance for adaptively routed traffic, while still allowing atomic operations to be idempotently applied directly to a user memory location.

First claim

What is claimed is:

1. A method for performing inter-process communication between nodes of a multi-node system, the method comprising:
defining, by a circuit, a window that represents a range of consecutively numbered packets, the window having a starting index number corresponding to the first packet of the range;
receiving, by the circuit, a connection initialization packet from a remote node;
sending, by the circuit, a response packet to the remote node including a maximum window size associated the window and a starting sequence number, wherein the response packet is configured to cause the remote node to assign sequence numbers to the plurality of packets sent therefrom with a starting sequence number set to the starting sequence number included in the response packet;
receiving, by the circuit, a plurality of packets from the remote node, each received packet having a unique sequence number;
determining, by the circuit, receipt of a consecutive set of received packets within the range of consecutively numbered packets, wherein the consecutive set of received packets has consecutive sequence numbers, and a first packet of the consecutive set has a sequence number corresponding to the starting index number of the window;
causing, by the circuit, the remote node to suspend transmitting packets before transmitting a packet with a sequence number that is equal to a max sequence number, the max sequence number being calculated as a sum of the starting sequence number and the maximum window size; and
transmitting, to the remote node by the circuit, an indication in response to determining receipt of the consecutive set of received packets.

2. The method of claim 1 , wherein the indication comprises a special acknowledgement packet, the special acknowledgment packet including a sequence number corresponding to the last packet of the consecutive set of received packets within the range.

3. The method of claim 2 , wherein the special acknowledgement packet is transmitted periodically to the remote node based on a timer.

4. The method of claim 1 , further comprising:
in response to determining receipt of the consecutive set of received packets, shifting the window such that the starting index number of the window corresponds to a packet associated with a sequence number beyond the last packet of the consecutive set.

5. The method of claim 1 , wherein a packet of the received plurality of packets comprises at least one memory operation, and wherein the method further comprises causing the at least one memory operation to be performed against a region of a memory.

6. The method of claim 5 , wherein the region of the memory comprises user-space memory.

7. The method of claim 5 , wherein the at least one memory operation includes an atomic operation.

8. The method of claim 1 , wherein the window size is at least 1000 packets.

9. The method of claim 1 , wherein determining receipt of the consecutive set of received packets within the range further comprises determining the consecutive set includes a minimum number of packets.

10. The method of claim 9 , wherein the minimum number of packets is a number less than a total number of consecutive packets represented by the window.

11. The method of claim 1 , wherein the circuit is integrated into a fabric interface circuit.

12. A system comprising:
a first node device comprising:
a memory;
circuitry configured to be communicatively coupled to a multi-node network, the circuitry being configured to:
define a window that represents a range of consecutively numbered packets, the window having a starting index number corresponding to the first packet of the range;
receive a connection initialization packet from a remote node;
transmit a response packet to the remote node including a maximum window size and a starting sequence number;
receive a plurality of packets from the remote node, each received packet having a unique sequence number;
determine receipt of a consecutive set of received packets within the range of consecutively numbered packets, wherein the consecutive set of received packets has consecutive sequence numbers, and a first packet of the consecutive set has a sequence number corresponding to the starting index number of the window;
suspend transmitting packets before transmitting a packet with a sequence number that is equal to a max sequence number, the max sequence number being calculated as a sum of the starting sequence number and the maximum window size; and
transmit to the remote node an indication in response to determining receipt of the consecutive set of received packets.

13. The system of claim 12 , wherein the indication comprises a special acknowledgement packet, the special acknowledgment packet including a sequence number corresponding to the last packet of the consecutive set of received packets within the range.

14. The system of claim 12 , wherein the circuitry of the first node device is further configured to:
in response to determining receipt of the consecutive set of received packets, shift the window such that the starting index number of the window corresponds to a packet associated with a sequence number beyond the last packet of the consecutive set.

15. The system of claim 12 , wherein the circuitry of the first node device comprises a network interface circuit.

16. A node device comprising:
a memory;
a network interface circuit to communicate on a multi-node network and to receive a plurality of packets from a remote node, each received packet having a unique sequence number;
means for defining a window that represents a range of consecutively numbered packets, the window having a starting index number corresponding to the first packet of the range;
means for receiving a connection initialization packet from the remote node;
means for transmitting a response packet to the remote node including a maximum window size and a starting sequence number;
means for determining receipt of a consecutive set of received packets within the range of consecutively numbered packets, wherein the consecutive set of received packets has consecutive sequence numbers, and a first packet of the consecutive set has a sequence number corresponding to the starting index number of the window;
means for suspending transmitting packets before transmitting a packet with a sequence number that is equal to a max sequence number, the max sequence number being calculated as a sum of the starting sequence number and the maximum window size; and
means for shifting the window such that the starting index number of the window corresponds to a packet associated with a sequence number beyond the last packet of the consecutive set in response to determining receipt of the consecutive set of received packets.

17. The node device of claim 16 , wherein the network interface circuit is further configured to transmit an indication to a remote node in response to determining receipt of the consecutive set of received packets.

18. The node device of claim 16 , wherein the indication comprises a special acknowledgement packet, the special acknowledgment packet including a sequence number corresponding to the last packet of the consecutive set of received packets within the range.

Assignees

• Intel Corp

Inventors

• Underwood Keith
• Giefer Charles
• Debbage Mark
• Brummel Karl P
• Miller Nathan
• Pirie Bruce

Classifications

• H04L69/22

  Parsing or analysis of headers · CPC title

• H04L1/1621

  Group acknowledgement, i.e. the acknowledgement message defining a range of identifiers, e.g. of sequence numbers · CPC title

• H04L47/27

  Evaluation or update of window size, e.g. using information derived from acknowledged [ACK] packets · CPC title

• H04L47/18

  End to end · CPC title

• H04L47/58

  Changing or combining different scheduling modes, e.g. multimode scheduling · CPC title