Method and apparatus for delivering MSI-X interrupts through non-transparent bridges to computing resources in PCI-express clusters

Patent metadata

Field	Value
Publication number	US-9465760-B2
Application number	US-201314083206-A
Country	US
Kind code	B2
Filing date	Nov 18, 2013
Priority date	Nov 18, 2013
Publication date	Oct 11, 2016
Grant date	Oct 11, 2016

Abstract

An apparatus for initialization. The apparatus includes a management I/O device controller for managing initialization of a plurality of I/O devices coupled to a PCI-Express (PCIe) fabric. The management I/O device controller is configured for receiving a request to register a target interrupt register address of a first worker computing resource, wherein the target interrupt register address is associated with a first interrupt generated by a first I/O device coupled to the PCIe fabric. A mapping module of the management I/O device controller is configured for mapping the target interrupt register address to a mapped interrupt register address of a domain in which the first I/O device resides. A translating interrupt register table includes a plurality of mapped interrupt register addresses in the domain that is associated with a plurality of target interrupt register addresses of a plurality of worker computing resources.

First claim

The invention claimed is:

1. An apparatus, comprising:
a management I/O device controller for managing initialization of a plurality of I/O devices coupled to a PCI Express (PCIe) fabric, wherein said management I/O device controller is configured for receiving a request to register a target interrupt register address of a first worker computing resource, wherein said target interrupt register address is associated with a first interrupt generated by a first I/O device coupled to said PCIe fabric;
a mapping module of said management I/O device controller configured for mapping said target interrupt register address to a mapped interrupt register address of a domain in which said first I/O device resides; and
a translating interrupt register table comprising a plurality of mapped interrupt register addresses in said domain that is associated with a plurality of target interrupt register addresses of a plurality of worker computing resources,
wherein said management I/O device controller is further configured for sending instructions to said first I/O device to register said mapped interrupt register address in association with said first interrupt in a corresponding I/O interrupt vector table of said first I/O device.

2. The apparatus of claim 1 , wherein said management I/O device controller is further configured for determining whether a vector entry is available within said I/O interrupt vector table for registering said first interrupt with said first I/O device and for sending instructions to register said mapped interrupt register address as a first vector in a first vector entry.

3. The apparatus of claim 1 , wherein said management I/O device controller is further configured for receiving information related to a size of a target interrupt register of said first worker computing resource including said target interrupt register address, and configured for allocating a first subset of virtual memory space in said translating interrupt register table controlled by said management I/O controller, wherein said first subset is at least equal to said size, and configured for mapping addresses in said target interrupt register to a plurality of mapped addresses in said virtual memory space.

4. The apparatus of claim 1 , wherein said management I/O device controller is further configured for sending an instruction to said I/O device to enable said first interrupt in said first I/O device, and configured for sending a notification to said first worker computing resource indicating that said first interrupt has been enabled in said first I/O device.

5. The apparatus of claim 1 , wherein said first interrupt is taken from a group consisting essentially of a message signal interrupt (MSI) interrupt, and an MSI-X interrupt.

6. A method for initialization, comprising:
receiving a request to register a target interrupt register address of a first worker computing resource, wherein said target interrupt register address is associated with a first interrupt generated by a first I/O device coupled to a PCI-Express (PCIe) fabric;
mapping said target interrupt register address to a mapped interrupt register address of a domain in which said first I/O device resides;
sending instructions to said first I/O device to register said mapped interrupt register address in association with said first interrupt in a corresponding I/O interrupt vector table of said first I/O device.

7. The method of claim 6 , further comprising:
sending mapping information mapping said mapped interrupt register address to said target interrupt register address to a first non-transparent bridge (NTB) coupling said first worker computing resource to said PCIe fabric.

8. The method of claim 6 , wherein said request is received at a management I/O device controller, wherein said management I/O device controller is coupled to said PCIe fabric to manage initialization of said plurality of I/O devices.

9. The method of claim 6 , wherein said sending instruction further comprises:
determining whether a vector entry is available within said I/O interrupt vector table for registering said first interrupt with said first I/O device and for storing said mapped interrupt register address as a first vector in said I/O interrupt vector table; and
sending said mapped interrupt register address to said first I/O device for storage in said available entry.

10. The method of claim 9 , further comprising:
determining whether said first I/O device can generate said first interrupt.

11. The method of claim 6 , further comprising:
sending an instruction to said first I/O device to enable said first interrupt in said first I/O device.

12. The method of claim 6 , further comprising:
sending a notification to said first worker computing resource indicating that said first interrupt has been enabled in said first I/O device.

13. The method of claim 6 , further comprising:
receiving information related to a size of target interrupt register of said first worker computing resource, wherein said target interrupt registers comprises a plurality of target interrupt register addresses;
allocating a first subset of virtual memory space in a mapped interrupt register comprising a plurality of mapped interrupt register addresses in said domain, wherein said mapped interrupt register is controlled by said management I/O controller, wherein said first subset is at least equal to said size; and
mapping addresses in said target interrupt register to a plurality of mapped addresses in said first subset of virtual memory space.

14. The method of claim 6 , wherein said first interrupt is taken from a group consisting essentially of a message signal interrupt (MSI) interrupt, and an MSI-X interrupt.

15. A system, comprising:
a plurality of worker computing resources;
a plurality of target interrupt registers associated with said plurality of worker computing resources;
a plurality of I/O devices located on said plurality of worker computing resources;
a PCI-Express (PCIe) fabric for enabling communication between said plurality of I/O devices and said plurality of worker computing resources all of which are coupled to said PCIe fabric;
a management I/O device controller configured for managing initialization of said plurality of I/O devices;
a translating interrupt register table comprising a plurality of mapped interrupt register addresses associated with a plurality of target interrupt register addresses of said plurality of worker computing resources;
wherein said management I/O device controller is configured for receiving a request to register a target interrupt register address of a first worker interrupt register of a first worker computing resource, wherein said target interrupt register address is associated with a first interrupt generated by a first I/O device coupled to said PCIe fabric; and
a mapping module of said management I/O device controller configured for mapping said target interrupt register address to a mapped interrupt register address, wherein said management I/O device controller is further configured for sending instructions to said first I/O device to register said mapped interrupt register address in association with said first interrupt in a corresponding I/O interrupt vector table of said first I/O device.

16. The system of claim 15 , further comprising:
a plurality of non-transparent bridges (NTBs) coupled to said plurality of worker computing resources, wherein a first NTB bridge is coupled to said management I/O device controller and is configured for sending said first interrupt originally delivered to said mapped inter

Assignees

• Futurewei Technologies Inc

Inventors

• Egi Norbert
• Lasater Robert
• Boyle Thomas
• Peters John
• Shi Guangyu

Classifications

• G06F13/4221

  being an input/output bus, e.g. ISA bus, EISA bus, PCI bus, SCSI bus · CPC title

• G06F13/404

  with address mapping · CPC title

• G06F13/4068

  Electrical coupling · CPC title

• G06F13/24 Primary

  using interrupt (G06F13/32 takes precedence) · CPC title

• G06F13/32 Primary

  using combination of interrupt and burst mode transfer · CPC title