Non-transparent bridge selection

Patent metadata

Field	Value
Publication number	US-12079154-B2
Application number	US-202318152257-A
Country	US
Kind code	B2
Filing date	Jan 10, 2023
Priority date	Jan 10, 2023
Publication date	Sep 3, 2024
Grant date	Sep 3, 2024

Abstract

A storage engine has a pair of compute nodes, each compute node having a separate PCIe root complex and attached memory. The PCIe root complexes are interconnected by multiple Non-Transparent Bridge (NTB) links. The NTB resources are unequally shared, such that host IO devices are required to use a first subset of the NTB links to implement memory access operations on the memory of the peer compute node, whereas storage software memory access operations are able to be implemented on all of the NTB links. A NTB link arbitration system arbitrates usage of the first and second subsets of NTB links by the storage software, to distribute subsets of the storage software memory access operations on peer memory to the first and second subsets of NTB links, while causing all host IO device memory access operations on peer memory to be implemented on the first set of NTB links.

First claim

What is claimed is:

1. A method of implementing non-transparent bridge selection in a storage engine including multiple unequally shared Non-Transparent Bridge (NTB) resources including a plurality of NTB links interconnecting a local compute node and a peer compute node within the storage engine, comprising:
implementing host Input/Output (IO) device memory access operations on a peer memory in a peer compute node by host IO devices residing on the local compute node, the host IO devices being connected to the peer memory by a first subset of the NTB links to a peer Peripheral Component Interconnect Express (PCIe) root complex in the peer compute node;
implementing storage software memory access operations on the peer memory in the peer compute node by storage software instantiated on the local compute node, the storage software being configured to access the peer memory by both the first subset of the NTB links to the peer PCIe root complex in the peer compute node and by a second subset of the NTB links to the peer PCIe root complex in the peer compute node;
arbitrating usage of the first subset of the NTB links and the second subset of the NTB links by the storage software, to cause a first subset of the storage software memory access operations on the peer memory to be implemented on the first subset of the NTB links and to cause a second subset of the storage software memory access operations on the peer memory to be implemented on the second subset of the NTB links, while causing all of the host IO device memory access operations on the peer memory to be implemented on the first subset of the NTB links.

2. The method of claim 1 , wherein the first subset of the NTB links comprise a first plurality of the NTB links; wherein the second subset of the NTB links comprise a second plurality of NTB links; and wherein the first subset of the NTB links does not include any NTB link included in the second subset of the NTB links.

3. The method of claim 1 , further comprising:
transmitting host IO device NTB link usage information associated with each host IO device memory access operation implemented on the peer memory to an NTB link arbitration system; and
transmitting storage software NTB link usage information associated with each storage software memory access operation implemented on the peer memory to the NTB link arbitration system.

4. The method of claim 3 , wherein arbitrating usage of the first subset of the NTB links and the second subset of the NTB links by the storage software comprises:
using the host IO device NTB link usage information and the storage software NTB link usage information to update NTB link data usage values of a NTB usage data structure, the NTB usage data structure including a data usage value for each of the NTB links; and
using the NTB link data usage values, by the NTB link arbitration system, to select a respective NTB link to be used to implement each respective storage software memory access operations on the peer memory.

5. The method of claim 4 , wherein the NTB link data usage values are exponential moving average NTB link usage values.

6. The method of claim 4 , wherein using the NTB link data usage values by the NTB link arbitration system comprises selecting the respective NTB link with a lowest respective NTB link data usage value.

7. A storage engine, comprising:
a local compute node having a local Peripheral Component Interconnect Express (PCIe) root complex connected to a local memory;
a peer compute node having a peer PCIe root complex connected to a peer memory;
a first subset of Non-Transparent Bridge (NTB) links interconnecting the local PCIe root complex of the local compute node with the peer PCIe root complex of the peer compute node, and a second subset of NTB links interconnecting the local PCIe root complex of the local compute node with the peer PCIe root complex of the peer compute node, the first subset of NTB links and the second subset of NTB links being mutually exclusive, such that the first subset of NTB links does not include any of the NTB links of the second subset of NTB links, and the second subset of NTB links does not include any of the NTB links of the first subset of NTB links;
a plurality of local host Input/Output (IO) devices on the local compute node, the local host IO devices being configured to implement host IO device memory operations on local memory via the local PCIe root complex and being configured to implement host IO device memory operations on the peer memory via the peer PCIe root complex using the first subset of NTB links;
storage software on the local compute node being configured to implement storage software memory operations on local memory via the local PCIe root complex and being configured to implement storage software memory operations on the peer memory via the peer PCIe root complex using both the first subset of NTB links and the second subset of NTB links; and
one or more computers and one or more storage devices storing instructions that are operable to implement an NTB link arbitration system, the instructions, when executed by the one or more computers, cause the NTB link arbitration system to perform operations comprising:
arbitrating usage of the first subset of NTB links and the second subset of NTB links by storage software, to cause a first subset of the storage software memory access operations on the peer memory to be implemented on the first subset of NTB links and to cause a second subset of the storage software memory access operations on the peer memory to be implemented on the second subset of NTB links, while causing all of the host IO device memory access operations on the peer memory to be implemented on the first subset of the NTB links.

8. The storage engine of claim 7 :
wherein the host IO devices are configured to transmit NTB link usage information associated with each host IO device memory access operation implemented on the peer memory to the NTB link arbitration system; and
wherein the storage software is configured to transmit NTB link usage information associated with each storage software memory access operation implemented on the peer memory to the NTB link arbitration system.

9. The storage engine of claim 8 , wherein the instructions, when executed by the one or more computers, further cause the NTB link arbitration system to perform operations comprising:
using the host IO device NTB link usage information and the storage software NTB link usage information to update NTB link data usage values of a NTB usage data structure, the NTB usage data structure including a data usage value for each of the NTB links; and
using the NTB link data usage values, by the NTB link arbitration system, to select a respective NTB link to be used to implement each respective storage software memory access operations on the peer memory.

10. The storage engine of claim 9 , wherein the NTB link data usage values are exponential moving average NTB link usage values.

11. The storage engine of claim 9 , wherein using the NTB link data usage values by the NTB link arbitration system comprises selecting the respective NTB link with a lowest respective NTB link data usage value.

12. A non-transitory tangible computer-readable medium storing software for implementing non-transparent bridge selection in a storage engine including multiple unequally shared Non-Transparent Bridge (NTB) resources including a plurality of NTB links interconnecting a local compute node and a peer compute node within the storage engine, comprising instructions executable by one or more computers which, upon such execution, cause the one or more computers to perform operations comprising:

Assignees

• Dell Products Lp

Inventors

• Krasner Jonathan
• Monserrat Ro
• Cartmell Jerome
• Mackintosh Thomas

Classifications

• G06F13/4009 Primary

  with data restructuring · CPC title

• G06F2213/0026

  PCI express · CPC title

• G06F13/4282

  on a serial bus, e.g. I2C bus, SPI bus (on daisy chain buses G06F13/4247) · CPC title

• G06F13/4027

  using bus bridges (G06F13/4022 takes precedence) · CPC title

• G06F13/4221 Primary

  being an input/output bus, e.g. ISA bus, EISA bus, PCI bus, SCSI bus · CPC title