Hierarchical/lossless packet preemption to reduce latency jitter in flow-controlled packet-based networks

Patent metadata

Field	Value
Publication number	US-10230665-B2
Application number	US-201314136293-A
Country	US
Kind code	B2
Filing date	Dec 20, 2013
Priority date	Dec 20, 2013
Publication date	Mar 12, 2019
Grant date	Mar 12, 2019

Abstract

Methods, apparatus, and systems for implementing hierarchical and lossless packet preemption and interleaving to reduce latency jitter in flow-controller packet-based networks. Fabric packets are divided into a plurality of data units, with data units for different fabric packets buffered in separate buffers. Data units are pulled from the buffers and added to a transmit stream in which groups of data units are interleaved. Upon receipt by a receiver, the groups of data units are separated out and buffered in separate buffers under which data units for the same fabric packets are grouped together. In one aspect, each buffer is associated with a respective virtual lane (VL), and the fabric packets are effectively transferred over fabric links using virtual lanes. VLs may have different levels of priority under which data units for fabric packets in higher-priority VLs may preempt fabric packets in lower-priority VLs. By transferring data units rather than entire packets, transmission of a packet can be temporarily paused in favor of a higher-priority packet. Multiple levels of preemption and interleaving in a nested manner are supported.

First claim

What is claimed is:

1. An apparatus, comprising circuitry and logic to:
divide respective ones of a plurality of fabric packets into a plurality different types of data units, including a first data unit comprising a head data unit, followed by a plurality of body data units, and ending with a last data unit comprising a tail data unit, each body data unit composed of a data payload and indicia composed of a single bit indicating it is a body data unit, each data unit having a predetermined fixed size, each fabric packet conveying data corresponding to a respective message;
generate a transmit stream of interleaved data units in which data units divided from a plurality of fabric packets are interleaved; and
transmit the transmit stream of interleaved data units outbound onto a link,
wherein the apparatus further comprises a transmit port including a transmit buffer having buffer space allocated to each of a plurality of virtual lane (VL) buffers, each VL buffer associated with a respective virtual lane, and the transmit port further includes circuitry and logic to:
buffer data units divided from a first fabric packet in a first VL buffer;
buffer data units divided from a second fabric packet in a second VL buffer;
add data units from the first VL buffer to the transmit stream of interleaved data units, the data units comprising a first portion of the data units divided from the first fabric packet;
interleave data units divided from the second fabric packet with the first fabric packet by adding data units from the second VL buffer to the transmit stream of interleaved data units, the data units comprising at least a first portion of the data units divided from the second fabric packet.

2. The apparatus of claim 1 , wherein the apparatus further includes circuitry and logic to:
associate the data units divided from respective ones of the plurality of fabric packets with a priority level assigned to the respective fabric packet; and
facilitate dynamic fabric packet preemption such that a stream of data units having a higher priority are interleaved into a stream of data units having a lower priority in the transmit stream of interleaved data units.

3. The apparatus of claim 2 , wherein the apparatus further includes circuitry and logic to:
facilitate multiple levels of fabric packet preemption under which transmission of data units with successively higher priority levels preempts transmission of data units in a nested manner having multiple levels of interleaving.

4. The apparatus of claim 2 , wherein the apparatus further includes a transmit buffer having buffer space allocated to a plurality of virtual lane (VL) buffers, respective ones of the plurality of VL buffers associated with a respective virtual lane, and the apparatus further includes circuitry and logic to:
assign a priority level for respective ones of the plurality of virtual lanes and associated VL buffers;
detect a virtual lane assigned to respective ones of the plurality of fabric packets;
buffer data units in a VL buffer based on the virtual lane assigned to the fabric packet from which the data units are divided;
detect that a first VL buffer has buffered data units ready to transmit;
pull data units for the first VL buffer and add the data units to the transmit stream of interleaved data units;
while adding data units from the first VL buffer,
detect that a second VL buffer having a higher priority level than the first VL buffer has buffered data units ready to transmit; and
in response thereto, preempt transmission of data units in the first VL buffer in favor of transmission of data units from the second VL buffer by,
pausing pulling of data units from the first VL buffer; and
pulling data units from the second VL buffer and adding the data units to the transmit stream of interleaved data units.

5. The apparatus of claim 4 , wherein the apparatus further includes circuitry and logic to:
assign the same priority level for the first virtual lane and a third virtual lane;
buffer data units divided from the first fabric packet in the first VL buffer and buffer data units divided from a third fabric packet in a third VL buffer associated with the third virtual lane;
pull data units for the first VL buffer and add the data units to the transmit stream of interleaved data units;
while adding data units from the first VL buffer,
detect a bubble in the availability of data units divided from the first fabric packet in the first VL buffer,
detect data units in the third VL buffer are ready to transmit; and
in response to detecting the bubble and data units in the third VL buffer are ready to transmit, interleaving transmission of data units in the third VL buffer in the transmit stream of interleaved data units by, pulling data units from the third VL buffer and adding the data units to the transmit stream of interleaved data units.

6. The apparatus of claim 1 , wherein the interleaved data units include interleaved data units divided from at least three fabric packets, and wherein the interleaved data units are interleaved in a nested fashion having at least two levels.

7. The apparatus of claim 1 , wherein the apparatus further includes circuitry and logic to:
continue to add data units from the second VL buffer to the stream of data units until the last data unit divided from the second fabric packet is added; and
resume adding data units from the first VL buffer to the transmit stream of interleaved data units until the last data unit divided from the first fabric packet is added.

8. The apparatus of claim 1 , wherein the apparatus further includes circuitry and logic to:
buffer data units divided from a third fabric packet in a third VL buffer;
interleave data units divided from the third fabric packet with the first and second fabric packets by adding data units from the third VL buffer to the transmit stream of interleaved data units after a first portion of the data units divided from the second fabric packet have been added to the transmit stream of interleaved data units, the data units comprising at least a first portion of the data units divided from the third fabric packet.

9. The apparatus of claim 1 , wherein the apparatus further includes circuitry and logic to:
track an order of the VL buffers from which data units are pulled and added to the transmit stream of interleaved data units to produce a transmit stream of interleaved data units having at least one nested level of interleaving.

10. The apparatus of claim 9 , wherein the order of the VL buffers from which data units are pulled and added to the transmit stream of interleaved data units is tracked through use of a VL stack and a VL register, wherein the VL register is used to store VL indicia identifying a current VL buffer data units are being pulled from, when a level of interleaving is added to the transmit stream of interleaved data units the VL of the VL buffer from which the data units are pulled is stored in the VL register and the prior VL indicia in the VL register is pushed onto a top of the VL stack, and when interleaving of that level of interleaving is completed the VL indicia on top of the VL stack is popped into the VL register.

11. The apparatus of claim 10 , wherein the apparatus further includes circuitry and logic to:
generate a control data unit comprising a VL marker including indicia use to identify a virtual lane to be associated with data units following the VL marker, wherein the VL register is updated with the VL marker indicia and the prior VL register indicia that is updated is pushed onto the VL stack; and
add the VL marker to the transmit stream of interleaved data units.

Assignees

• Intel Corp

Inventors

• Lovett Thomas D
• Cheng Albert
• Birrittella Mark S
• Kunz James
• Rimmer Todd

Classifications

• H04L47/621

  Individual queue per connection or flow, e.g. per VC · CPC title

• H04L1/0071 Primary

  Use of interleaving (interleaving per se H03M13/27) · CPC title

• H04L47/6275

  based on priority · CPC title

• H04L25/14

  Channel dividing arrangements {, i.e. in which a single bit stream is divided between several baseband channels and reassembled at the receiver} · CPC title

• H04L47/821 Primary

  Prioritising resource allocation or reservation requests · CPC title