Optimized credit return mechanism for packet sends

Patent metadata

Field	Value
Publication number	US-9792235-B2
Application number	US-201615285825-A
Country	US
Kind code	B2
Filing date	Oct 5, 2016
Priority date	Jun 26, 2014
Publication date	Oct 17, 2017
Grant date	Oct 17, 2017

Abstract

Method and apparatus for implementing an optimized credit return mechanism for packet sends. A Programmed Input/Output (PIO) send memory is partitioned into a plurality of send contexts, each comprising a memory buffer including a plurality of send blocks configured to store packet data. A storage scheme using FIFO semantics is implemented with each send block associated with a respective FIFO slot. In response to receiving packet data written to the send blocks and detecting the data in those send blocks has egressed from a send context, corresponding freed FIFO slots are detected, and a lowest slot for which credit return indicia has not be returned is determined. The highest slot in a sequence of freed slots from the lowest slot is then determined, and corresponding credit return indicia is returned. In one embodiment an absolute credit return count is implemented for each send context, with an associated absolute credit sent count tracked via software that writes to the PIO send memory, with the two absolute credit counts used for flow control.

First claim

What is claimed is:

1. A method comprising:
partitioning memory space in a Programmed Input/Output (PIO) send memory into a plurality of send contexts, each comprising a memory buffer including a plurality of send blocks configured to store packet data;
implementing a storage scheme using First-in, First-out (FIFO) semantics for each send context under which each send block occupies a respective FIFO slot in a FIFO buffer having a FIFO order and data for a given packet is stored in one or more send blocks occupying one or more respective sequential FIFO slots in a FIFO order;
receiving packet data written to send blocks out of order such that for at least a portion of packets send blocks are filled with packet data in a different order than the FIFO order;
egressing send blocks containing filled packet data;
in response to detecting one or more send blocks have been egressed from a send context,
detecting one or more FIFO slots that have been freed;
detecting when one or more FIFO slots that have been freed occupies a lowest slot in the FIFO order for the send context for which credit return indicia has not been returned; and
returning credit return indicia from which a highest FIFO slot of a sequence of freed slots in the FIFO order including the lowest slot for which credit information has not been returned can be identified.

2. The method of claim 1 , further comprising:
maintaining, for each send context, an absolute credit sent count identifying an aggregate number of send blocks that have been or will be filled via PIO write instructions that have been dispatched for execution since the absolute credit sent count for the send context was initialized;
receiving an absolute credit return count for a send context; and
comparing the absolute credit sent count for the send context to the absolute credit returned count for the send context to determine whether additional packet data can be written to the send context.

3. The method of claim 2 , further comprising:
implementing physical circular FIFOs for storing send blocks in a send context;
implementing logical circular FIFOs for tracking the absolute credit return count and the absolute credit sent count for the send context, wherein the size of the logical circular FIFO is larger than the size of the physical circular FIFO.

4. The method of claim 2 , further comprising:
dispatching PIO write instructions for writing data to a send context in the PIO send memory to a processor in a circular FIFO order, wherein the circular FIFO has a number of FIFO slots that matches a number of send blocks for the send context, and wherein each PIO write instruction defines a PIO send memory address corresponding to a respective send block; and
pausing dispatch of PIO write instructions to the processor if a difference between the absolute credit sent count and the absolute credit return count reaches the number of FIFO slots.

5. The method of claim 2 , further comprising:
defining a minimum credit return threshold; and
returning an absolute credit return count for a send context if a difference between the absolute credit return count and a most-recently returned absolute credit return count for the send context would meet or exceed the minimum credit return threshold; otherwise
waiting to return an absolute credit return count until the difference between the absolute credit return count and the most-recently returned absolute credit return count for the send context would meet or exceed the minimum credit return threshold.

6. The method of claim 1 , further comprising preventing any send blocks storing data for a given packet from egressing until all packet data for the packet has been written to the PIO send memory.

7. An apparatus, comprising:
an input/output (IO) interface;
a transmit engine coupled to the IO interface and including,
a Programmed Input/Output (PIO) send memory;
an egress block, operatively coupled to the PIO send memory; and
circuitry and logic to,
partition the PIO send memory into a plurality of send contexts, each comprising a plurality of sequential send blocks;
implement a storage scheme using First-in, First-out (FIFO) semantics for each send context under which each send block occupies a respective FIFO slot in a FIFO buffer having a FIFO order and data for a given packet is stored in one or more send blocks occupying one or more respective sequential FIFO slots in a FIFO order;
receive packet data written to send blocks out of order such that for at least a portion of packets send blocks are filled with packet data in a different order than the FIFO order;
egress send blocks containing filled packet data;
in response to detecting one or more send blocks have been egressed from a send context,
detect one or more FIFO slots that have been freed; and
detect when one or more FIFO slots that have been freed occupies a lowest slot in the FIFO order for the send context for which credit return indicia has not been returned; and
return credit return indicia from which a highest FIFO slot of a sequence of freed slots in the FIFO order including the lowest slot for which credit information has not been returned can be identified.

8. The apparatus of claim 7 , wherein the credit return indicia is an absolute credit return count comprising an aggregate number of send blocks in the FIFO order for the send context that have been freed since the absolute credit return count for the send context was initialized.

9. The apparatus of claim 8 , wherein the transmit engine further includes logic to: implement physical circular FIFOs for storing send blocks in the send contexts; and
implement logical circular FIFOs for tracking the absolute credit return count for each send context, wherein the size of the logical circular FIFOs is the same for all send contexts while the size of at least two send contexts and their associated physical circular FIFOs are different.

10. The apparatus of claim 8 , wherein the transmit engine further includes logic to: return an absolute credit return count for a send context if a difference between the absolute credit return count and a most-recently returned absolute credit return count for the send context would meet or exceed a minimum credit return threshold; otherwise
wait to return an absolute credit return count until the difference between the absolute credit return count and the most-recently returned absolute credit return count for the send context would meet or exceed the minimum credit return threshold.

11. The apparatus of claim 7 , wherein the transmit engine further includes logic to: concurrently detect freeing of FIFO slots in a plurality of send contexts and determine a plurality of absolute credit counts including an absolute credit count for each of the multiple send contexts among the plurality of send contexts; and
return the plurality of absolute credit return counts via a single credit return message.

12. The apparatus of claim 7 , wherein the IO interface comprises a first (PCIe) interface, the apparatus further comprising:
a processor, having a plurality of processor cores supporting out of order execution and including a memory interface, at least one store buffer, and a second PCIe interface coupled to the first PCIe interface via a PCIe interconnect; further including circuitry and logic to,
receive sequences of PIO write instructions to write packet data for respective packets stored in a memory when coupled to the memory interface to the PIO send memory;
execute the sequences of PIO write instructions as an instruction thread on a processor core, wherein execution of PIO write instructions cause data to be written t

Assignees

• Intel Corp

Inventors

• Debbage Mark
• Mutha Yatin M

Classifications

• G06F2213/0026

  PCI express · CPC title

• G06F15/7807

  System on chip, i.e. computer system on a single chip; System in package, i.e. computer system on one or more chips in a single package · CPC title

• G06F13/128 Primary

  for dedicated transfers to a network (for protocol converters G06F13/387) · CPC title

• G06F2205/067

  Bidirectional FIFO, i.e. system allowing data transfer in two directions · CPC title

• G06F13/1673 Primary

  using buffers · CPC title