Optimized credit return mechanism for packet sends

Patent metadata

Field	Value
Publication number	US-9984020-B2
Application number	US-201715784607-A
Country	US
Kind code	B2
Filing date	Oct 16, 2017
Priority date	Jun 26, 2014
Publication date	May 29, 2018
Grant date	May 29, 2018

Abstract

Method and apparatus for implementing an optimized credit return mechanism for packet sends. A Programmed Input/Output (PIO) send memory is partitioned into a plurality of send contexts, each comprising a memory buffer including a plurality of send blocks configured to store packet data. A storage scheme using FIFO semantics is implemented with each send block associated with a respective FIFO slot. In response to receiving packet data written to the send blocks and detecting the data in those send blocks has egressed from a send context, corresponding freed FIFO slots are detected, and a lowest slot for which credit return indicia has not be returned is determined. The highest slot in a sequence of freed slots from the lowest slot is then determined, and corresponding credit return indicia is returned. In one embodiment an absolute credit return count is implemented for each send context, with an associated absolute credit sent count tracked via software that writes to the PIO send memory, with the two absolute credit counts used for flow control.

First claim

What is claimed is:

1. A method comprising:
partitioning memory space in a Programmed Input/Output (PIO) send memory into a plurality of send contexts, each send context associated with a memory buffer including a plurality of send blocks configured to store packet data;
implementing a storage scheme using a First-in, First-out (FIFO) scheme for each send context under which each send block occupies a respective FIFO slot in a FIFO buffer having a FIFO order and data for a given packet is stored in one or more send blocks occupying one or more respective sequential FIFO slots in a FIFO order;
filling send blocks in a send context with out of order packet data;
egressing send blocks containing filled packet data from the send context;
in response to detecting one or more send blocks have been egressed from the send context,
detecting one or more FIFO slots that have been freed;
detecting when one or more FIFO slots that have been freed occupies a lowest slot in the FIFO order for the send context for which credit return indicia has not been returned; and
returning credit return indicia from which a highest FIFO slot of a sequence of freed slots in the FIFO order including the lowest slot for which credit information has not been returned can be identified.

2. The method of claim 1 , further comprising:
maintaining, for each send context, an absolute credit sent count identifying an aggregate number of send blocks that have been or will be filled via PIO write instructions that have been dispatched for execution since the absolute credit sent count for the send context was initialized;
receiving an absolute credit return count for a send context; and
comparing the absolute credit sent count for the send context to the absolute credit returned count for the send context to determine whether additional packet data can be written to the send context.

3. The method of claim 2 , further comprising:
implementing send memory wrap-around FIFOs for storing send blocks in a send context;
implementing host memory wrap-around FIFOs for tracking the absolute credit return count and the absolute credit sent count for the send context, wherein the size of the host memory wrap-around FIFO is larger than the size of the send memory wrap-around FIFO.

4. The method of claim 2 , further comprising:
dispatching PIO write instructions for writing data to a send context in the PIO send memory to a processor in a wrap-around FIFO order, wherein the wrap-around FIFO has a number of FIFO slots that matches a number of send blocks for the send context, and wherein each PIO write instruction defines a PIO send memory address corresponding to a respective send block; and
pausing dispatch of PIO write instructions to the processor if a difference between the absolute credit sent count and the absolute credit return count reaches the number of FIFO slots.

5. The method of claim 2 , further comprising:
defining a minimum credit return threshold; and
returning an absolute credit return count for a send context if a difference between the absolute credit return count and a most-recently returned absolute credit return count for the send context would meet or exceed the minimum credit return threshold; otherwise
waiting to return an absolute credit return count until the difference between the absolute credit return count and the most-recently returned absolute credit return count for the send context would meet or exceed the minimum credit return threshold.

6. The method of claim 1 , further comprising preventing any send blocks storing data for a given packet from egressing until all packet data for the packet has been written to the PIO send memory.

7. An apparatus, comprising:
an input/output (IO) interface;
a transmit engine coupled to the IO interface and including,
a send memory;
an egress block, operatively coupled to the send memory; and
circuitry to,
partition the send memory into a plurality of send contexts, each comprising a plurality of sequential send blocks;
implement a First-in, First-out (FIFO) storage scheme for each send context under which each send block occupies a respective FIFO slot in a FIFO buffer having a FIFO order and data for a given packet is stored in one or more send blocks occupying one or more respective sequential FIFO slots in a FIFO order;
receive packet data written to send blocks out of order such that for at least a portion of packets send blocks are filled with packet data in a different order than the FIFO order;
egress send blocks containing filled packet data;
in response to detecting one or more send blocks have been egressed from a send context,
detect one or more FIFO slots that have been freed; and
detect when the one or more FIFO slots that have been freed occupies a lowest slot in the FIFO order for the send context for which credit return indicia has not been returned; and
return credit return indicia from which a highest FIFO slot of a sequence of freed slots in the FIFO order including the lowest slot for which credit information has not been returned can be identified.

8. The apparatus of claim 7 , wherein the credit return indicia is an absolute credit return count comprising an aggregate number of send blocks in the FIFO order for the send context that have been freed since the absolute credit return count for the send context was initialized.

9. The apparatus of claim 8 , wherein the transmit engine further includes circuitry to:
implement send memory wrap-around FIFOs for storing send blocks in the send contexts; and
implement host memory wrap-around FIFOs for tracking the absolute credit return count for each send context, wherein the size of the host memory wrap-around FIFOs is the same for all send contexts while the size of at least two send contexts and their associated send memory wrap-around FIFOs are different.

10. The apparatus of claim 8 , wherein the transmit engine further includes circuitry to:
return an absolute credit return count for a send context if a difference between the absolute credit return count and a most-recently returned absolute credit return count for the send context would meet or exceed a minimum credit return threshold; otherwise
wait to return an absolute credit return count until the difference between the absolute credit return count and the most-recently returned absolute credit return count for the send context would meet or exceed the minimum credit return threshold.

11. The apparatus of claim 7 , wherein the transmit engine further includes circuitry to:
concurrently detect freeing of FIFO slots in a plurality of send contexts and determine a plurality of absolute credit counts including an absolute credit count for each of the multiple send contexts among the plurality of send contexts; and
return the plurality of absolute credit return counts via a single credit return message.

12. The apparatus of claim 7 , wherein the IO interface comprises a first (PCIe) interface, the apparatus further comprising:
a processor, having a plurality of processor cores supporting out of order execution and including a memory interface, at least one store buffer, and a second PCIe interface coupled to the first PCIe interface via a PCIe interconnect, further including circuitry to,
receive sequences of write instructions to write packet data for respective packets stored in a memory when coupled to the memory interface to the send memory;
execute the sequences of write instructions as an instruction thread on a processor core, wherein execution of write instructions cause data to be written to store units in a store buffer, the store units grouped into store blocks comprisin

Assignees

• Intel Corp

Inventors

• Debbage Mark
• Mutha Yatin M

Classifications

• G06F2213/36

  Arbitration · CPC title

• G06F2213/0026

  PCI express · CPC title

• G06F13/128 Primary

  for dedicated transfers to a network (for protocol converters G06F13/387) · CPC title

• G06F13/4282

  on a serial bus, e.g. I2C bus, SPI bus (on daisy chain buses G06F13/4247) · CPC title

• G06F2213/2806

  Space or buffer allocation for DMA transfers · CPC title