Dynamic assignment of groups of resources in a peripheral component interconnect express network

Patent metadata

Field	Value
Publication number	US-9858228-B2
Application number	US-201514822672-A
Country	US
Kind code	B2
Filing date	Aug 10, 2015
Priority date	Aug 10, 2015
Publication date	Jan 2, 2018
Grant date	Jan 2, 2018

Abstract

Systems and methods for analyzing a PCIe network using a graph-theory based analysis are disclosed. A management CPU is coupled to the root complex of the PCIe system and is operable to survey potential CPU-resource combinations in a PCIe system and assign a group of PCIe resources to a CPU. A first switch and a second switch are coupled to the root node, and a first CPU and a first group of PCIe resources are coupled to the first switch. The management CPU assigns a group of PCIe resources to a CPU based on the isolation of the first and second CPUs or a distance between the first and second CPUs and the groups of PCIe resources. According to some embodiments, for potential pairs of devices and NTB/CPUs, the distance between components is assessed, possible alternative paths are identified, and the isolation of the pair is determined.

First claim

What is claimed is:

1. An apparatus for dynamically assigning groups of resources in a PCIe system, comprising:
a management CPU coupled to a root complex of the PCIe system and operable to survey potential CPU-resource combinations in the PCIe system and assign a group of PCIe resources to a CPU;
a first switch and a second switch coupled to the root complex;
a first CPU and a first group of PCIe resources coupled to the first switch; and
a second CPU and a second group of PCIe resources coupled to the second switch, wherein a first path from the first CPU to the first group of PCIe resource comprises a first non-transparent bridge (NTB) and does not traverse the root complex, a second path from the second CPU to the second group of PCIe resource comprises a second NTB and does not traverse the root complex, the first path and second path are isolated from each other, and the management CPU is configured to assign the first or second group of PCIe resources to the first or second CPU based on at least one of a degree of isolation of the first and second CPUs and the first and second groups of PCIe resources and a distance between the first and second CPUs and the first and second groups of PCIe resources;
wherein the management CPU is further configured to:
create a first shortcut between the first CPU and the first group of PCIe resources using the first NTB;
create a second shortcut between the second CPU and the second group of PCIe resources using the second NTB;
calculate current potential traffic of the first and second paths based on current device assignments;
assign a third group of PCIe resources using the first or second path based on the current potential traffic when the current potential traffic of the first path is significantly different than the current potential traffic of the second path;
monitor current actual traffic over the first and second paths;
assign the third group of PCIe resources using the first or second path based on the current actual traffic when the current potential traffic of the first path is not different than the current potential traffic of the second path; and
assign the third group of PCIe resources using the first or second path based on a total number of hops along the first and second paths when the current actual traffic of the first path is not different than the current actual traffic of the second path.

2. A method for dynamically assigning a group of PCIe resources to a CPU in a PCIe system comprising:
identifying a first path and a second path between the CPU and the group of PCIe resources;
calculating potential traffic for the first and second paths based on device assignments;
monitoring actual traffic over the first and second paths; and
assigning the group of PCIe resources to the CPU using the first or second path based on:
the potential traffic when a first difference between the potential traffic of the first path and the potential traffic of the second path is greater than a first traffic threshold;
the actual traffic when a second difference between the potential traffic of the first path and the potential traffic of the second path is less than a second traffic threshold; and
the total number of hops along the first and second paths when a third difference between the actual traffic of the first path and the actual traffic of the second path is less than a third traffic threshold and a fourth difference between the potential traffic of the first path and the potential traffic of the second path is less than a fourth traffic threshold.

3. The method of claim 2 , wherein identifying a first path and a second path comprises:
creating a shortcut in the PCIe using an NTB; and
traversing a path from a first end of the NTB to a second end of the NTB to identify a cycle comprising the shortcut, wherein the cycle further comprises at least one of the first path and the second path.

4. The method of claim 2 , wherein calculating potential traffic is based on a maximum bandwidth of a device of the group of PCIe resources.

5. The method of claim 2 , wherein the CPU comprises an NTB.

6. The method of claim 2 , wherein the monitoring actual traffic is performed by a management CPU.

7. The method of claim 2 , further comprising tracking PCIe resource assignments related to the CPU and releasing a first PCIe resource assignment when the first PCIe resource assignment is no longer used by the CPU.

8. The method of claim 7 , further comprising reassigning the group of PCIe resources to a second CPU based on a change in a traffic condition, wherein the reassigning improves load balance.

9. A method for dynamically assigning a device to a CPU in a PCIe system comprising:
receiving a resource request comprising a plurality of device types;
calculating a first device type score for devices of a first device type associated with a first CPU;
calculating a second device type score for devices of a second device type associated with the first CPU;
calculating a first total CPU score by adding the first and second device type scores to a first CPU score associated with the first CPU;
calculating a third device type score for devices of a third device type associated with a second CPU;
calculating a fourth device type score for devices of a fourth device type associated with the second CPU;
calculating a second total CPU score by adding the third and fourth device type scores to a second CPU score associated with the second CPU; and
assigning the device to the first or second CPU based on the first and second total CPU scores, wherein the device is accessed using an NTB.

10. The method of claim 9 , further comprising calculating a plurality of device scores for a plurality of devices of the first, second, third, and fourth device types, and using the highest device score as the device type score for the respective device type.

11. The method of claim 10 , wherein the resource request further comprises a CPU capability requirement, and CPUs that do not satisfy the CPU capability requirement are excluded.

12. The method of claim 11 , wherein the CPU capability requirement comprises a number of CPU cores.

13. The method of claim 12 , wherein calculating the device scores is based on at least one of available bandwidth and available resources.

14. The method of claim 13 , wherein calculating the device scores is further based on a number of current assignments of an associated device.

15. The method of claim 14 , wherein calculating the device scores is further based on a degree of isolation between the associated device and the CPU.

16. The method of claim 15 , wherein calculating the device scores is further based on a number of hops between the associated device and the CPU.

17. The method of claim 16 , further comprising adding a bonus to the first, second, third, or fourth device score if the associated device is a high performance device.

18. The method of claim 17 , wherein the plurality of device scores are further based on an amount of current traffic associated with the devices.

Assignees

• Futurewei Technologies Inc

Inventors

• Lasater Robert

Classifications

• G06F13/4022 Primary

  using switching circuits, e.g. switching matrix, connection or expansion network (G06F13/4009 takes precedence) · CPC title

• G06F13/4282

  on a serial bus, e.g. I2C bus, SPI bus (on daisy chain buses G06F13/4247) · CPC title