Modular switched fabric for data storage systems

Patent metadata

Field	Value
Publication number	US-10503618-B2
Application number	US-201715494673-A
Country	US
Kind code	B2
Filing date	Apr 24, 2017
Priority date	Jun 23, 2014
Publication date	Dec 10, 2019
Grant date	Dec 10, 2019

Abstract

Systems, methods, apparatuses, and software for data storage systems are provided herein. In one example, a data storage system is provided that includes a plurality of storage drives each comprising a Peripheral Component Interconnect Express (PCIe) interface, and configured to store data and retrieve the data stored on associated storage media responsive to storage operations. The data storage system includes one or more processing modules comprising one or more processors communicatively coupled to the plurality of storage drives over a PCIe fabric comprised of one or more PCIe switches. The processors are configured to share a PCIe address space associated with the PCIe fabric for transfer of the storage operations to appropriate ones of the processors that manage ones of the plurality of storage drives.

First claim

What is claimed is:

1. A data storage system, comprising:
a plurality of storage drives each comprising a Peripheral Component Interconnect Express (PCIe) interface, and configured to store data and retrieve the data stored on associated storage media responsive to storage operations;
a plurality of processors communicatively coupled to network interfaces and the plurality of storage drives over a PCIe fabric comprised of one or more PCIe switches, wherein the one or more plurality of processors are configured to share a PCIe address space associated with the PCIe fabric, with each of the plurality of processors allocated to at least a corresponding portion of the PCIe address space;
each of the network interfaces configured to be associated with a different Internet Protocol (IP) address, and at least two of the network interfaces configured to be associated with an Internet Small Computer System Interface (iSCSI) portal group, wherein the network interfaces are configured to receive the storage operations over associated network links, and transfer ones of the storage operations to associated ones of the plurality of processors over the PCIe fabric; and
a first processor associated with a first network interface and the iSCSI portal group configured to identify data packets received over the first network interface as comprising a first storage operation, process the first storage operation against allocation information related to the PCIe address space to determine the first storage operation is associated with at least one of the plurality of storage drives managed by a second processor, and responsively transfer the first data packets for handling of the first storage operation by the second processor as if the data packets were received by a second network interface associated with the second processor and the iSCSI portal group.

2. The data storage system of claim 1 , comprising:
the first processor configured to receive the first storage operation and identify a portion of the PCIe address space associated with the first storage operation; and
when the portion of the PCIe address space is associated with the second processor, the first processor configured to transfer the first storage operation for delivery to the second processor over the PCIe fabric for handling of the first storage operation with the at least one of the plurality of storage drives assigned to the second processor.

3. The data storage system of claim 1 , comprising:
a front end portion of the PCIe fabric formed among at least a first one of the PCIe switches that communicatively couples the plurality of processors to each other; and
a back end portion of the PCIe fabric formed among at least a second one of the PCIe switches that communicatively couples the plurality of processors to the plurality of storage drives.

4. The data storage system of claim 1 , comprising:
a unified PCIe fabric formed by the PCIe fabric among the one or more PCIe switches that communicatively couples the plurality of processors to each other, and communicatively couples the plurality of processors to the plurality of storage drives.

5. The data storage system of claim 1 , comprising:
at least one of the PCIe switches configured to interconnect at least one processor of the data storage system with another data storage system over at least one external PCIe link.

6. The data storage system of claim 5 , comprising:
the at least one external PCIe link comprising PCIe signaling carried over Serial Attached Small Computer Serial Interconnect (SAS) connectors and cabling to interconnect the data storage system to the other data storage system.

7. The data storage system of claim 1 , comprising:
at least one of the plurality of processors configured to present an associated one or more storage drives as at least one NVM Express (NVMe) virtual logical unit number (VLUN) over a PCIe interface that communicatively couples the at least one of the plurality of processors and a computing system external to the data storage system; and
the at least one of the plurality of processors configured to receive NVMe storage transactions transferred by the computing system over the PCIe interface and handle the NVMe storage transactions for storage of data on the associated one or more storage drives.

8. A method of operating a data storage system, the method comprising:
in a plurality of storage drives, storing data and retrieving the data stored on associated storage media responsive to storage operations received over a Peripheral Component Interconnect Express (PCIe) interface;
in a plurality of processors communicatively coupled to network interfaces and the plurality of storage drives over a PCIe fabric comprised of one or more PCIe switches, sharing a PCIe address space associated with the PCIe fabric among the plurality of processors, with each of the plurality of processors allocated to at least a corresponding portion of the PCIe address space;
associating each of the network interfaces with a different Internet Protocol (IP) address and associating at least two of the network interfaces with an Internet Small Computer System Interface (iSCSI) portal group; and
in a first processor associated with a first network interface and the iSCSI portal group, identifying data packets received over the first network interface as comprising a first storage operation, processing the first storage operation against allocation information related to the PCIe address space to determine the first storage operation is associated with at least one of the plurality of storage drives managed by a second processor, and responsively transferring the first data packets for handling of the first storage operation by the second processor as if the data packets were received by a second network interface associated with the second processor and the iSCSI portal group.

9. The method of claim 8 , further comprising:
in the first processor, receiving the first storage operation and identifying a portion of the PCIe address space associated with the first storage operation; and
when the portion of the PCIe address space is associated with the second processor, in the first processor, transferring the first storage operation for delivery to the second processor over the PCIe fabric for handling of the first storage operation with the at least one of the plurality of storage drives assigned to the second processor.

10. The method of claim 8 , wherein a front end data plane is formed among at least a first one of the PCIe switches that communicatively couples the plurality of processors to each other; and wherein a back end data plane is formed among at least a second one of the PCIe switches that communicatively couples the plurality of processors to the plurality of storage drives.

11. The method of claim 8 , wherein a unified data plane is formed among the one or more PCIe switches that communicatively couples the plurality of processors to each other, and communicatively couples the plurality of processors to the plurality of storage drives.

12. The method of claim 8 , further comprising:
in at least one of the PCIe switches, interconnecting at least one of processors of the data storage system with one or more processors of another data storage system over at least one external PCIe link, wherein the at least one external PCIe link comprises PCIe signaling carried over Serial Attached Small Computer Serial Interconnect (SAS) connectors and cabling to interconnect the data storage system to the other data storage system.

13. The method of claim 8 , further comprising:
in at least one of the plurality of processors, present

Assignees

• Liqid Inc

Inventors

• Breakstone Jason
• Long Christopher R
• Cannata James Scott

Classifications

• G06F13/4022

  using switching circuits, e.g. switching matrix, connection or expansion network (G06F13/4009 takes precedence) · CPC title

• G06F3/0685

  Hybrid storage combining heterogeneous device types, e.g. hierarchical storage, hybrid arrays · CPC title

• G06F11/2017

  where memory access, memory control or I/O control functionality is redundant (redundant communication control functionality G06F11/2005; redundant storage control functionality G06F11/2089) · CPC title

• G06F13/4282

  on a serial bus, e.g. I2C bus, SPI bus (on daisy chain buses G06F13/4247) · CPC title

• G06F11/2012

  and using different communication protocols · CPC title