PCIe lane aggregation over a high speed link

Patent metadata

Field	Value
Publication number	US-10235318-B2
Application number	US-201715812493-A
Country	US
Kind code	B2
Filing date	Nov 14, 2017
Priority date	Aug 9, 2013
Publication date	Mar 19, 2019
Grant date	Mar 19, 2019

Abstract

A method of operating a computer network system configured with disaggregated inputs/outputs. This system can be configured in a leaf-spine architecture and include a router coupled to a network source, a plurality of core switches coupled to the router, a plurality of aggregator switches coupled to each of the plurality of core switches, and a plurality of rack modules coupled to each of the plurality of aggregator switches. Each of rack modules can include an I/O appliance with a downstream aggregator module, a plurality of server devices each with PCIe interfaces, and an upstream aggregator module that aggregates each of the PCIe interfaces. A high-speed link can be configured between the downstream and upstream aggregator modules via aggregation of many serial lanes to provide reliable high speed bit stream transport over long distances, which allows for better utilization of resources and scalability of memory capacity independent of the server count.

First claim

What is claimed is:

1. A method of operating a data center system having a computer network system coupled to a network; the computer network system having an I/O appliance, a plurality of service devices coupled to the I/O appliance, and an upstream aggregating silicon photonics device coupled to the plurality of service devices; the method comprising:
receiving, by a plurality of optical ports of the I/O appliance numbered from 1 to N, a data stream;
aggregating, by a downstream aggregating silicon photonics device of the I/O appliance provided on each of the plurality of optical ports, the data stream;
storing, by a Solid-State Drive (SSD) of the I/O appliance coupled to each of the optical ports, the data stream;
configuring, by a Network Interface Controller (NIC) interface of the I/O appliance coupled to each of the optical ports, the data stream;
wherein each of the NIC interfaces is coupled to a top of rack switch and a plurality of spine switches are coupled to the top of rack switches;
processing, by a network processor of the I/O appliance, the data stream;
storing, by a memory storage device of each of the plurality of service devices, the data stream;
processing, by a Central Processing Unit (CPU) of each of the plurality of server devices coupled to the memory storage device, the data stream;
configuring, by a Peripheral Component Interconnect Express (PCIe) interface configured with each of the CPU devices, the data stream; and
aggregating, by the upstream aggregating silicon photonics device, each of the PCIe interfaces to aggregate the data stream.

2. The method of claim 1 wherein aggregating each of the PCIe interfaces includes aggregating via a twisted pair configuration between the PCIe interfaces and the upstream silicon photonics device.

3. The method of claim 1 wherein configuring, by the PCIe interface, the data stream includes communicating at a data rate of 8 Gbps.

4. The method of claim 1 wherein configuring, by the PCIe interface, the data stream includes configuring the data stream according to a PAM format or a PAM-4 format.

5. The method of claim 1 wherein the I/O appliance is provided on a top rack spatial location.

6. The method of claim 1 wherein the downstream aggregating silicon photonics device and the upstream aggregating silicon photonics device are configured in a 1-to-1 mapping.

7. The method of claim 1 further comprising communicating, by a high-speed link configured between the downstream aggregating silicon photonics device and the upstream aggregating silicon photonics device, the data stream.

8. The method of claim 7 wherein communicating, by the high-speed link, the data stream includes communicating at a transfer rate of greater than 24 Gbps or greater than 36 Gbps.

9. The method of claim 7 further comprising converting, by a pair of optical converters coupled by an optical fiber configured between the upstream aggregating silicon photonics device and the downstream aggregating silicon photonics device, the data stream.

10. A method of operating a data center system having a computer network system configured in a leaf-spine architecture; the computer network system having a router coupled to a network source, a plurality of core switches coupled to the router, a plurality of aggregator switches coupled to each of the plurality of core switches, and a plurality of rack modules coupled to each of the plurality of aggregator switches, each of the rack modules including an Input/Output (I/O) appliance, a plurality of server devices coupled to the I/O appliance, and an upstream aggregator module coupled to the plurality of service devices; the method comprising:
receiving, by a plurality of optical ports of the I/O appliance numbered from 1 to N, a data stream;
routing, by a routing element of the I/O appliance coupled to the plurality of optical ports, the data stream;
configuring, by a Network Interface Controller (NIC) interface of the I/O appliance coupled to the routing element, the data stream;
aggregating, by a downstream aggregator module of the I/O appliance coupled to the NIC interface, the data stream;
wherein each of the NIC interfaces is coupled to a top of rack switch and a plurality of spine switches are coupled to the top of rack switches;
processing, by a network processor of the I/O appliance, the data stream;
configuring, by a memory storage controller of the I/O appliance coupled to the downstream aggregator module, the data stream;
storing, by a memory storage device of each of the plurality of server devices, the data stream;
processing, by a Central Processing Unit (CPU) device of each of the plurality of server devices coupled to the memory storage device, the data stream;
wherein each of the CPU devices is coupled to one or more memory modules;
configuring, by a Peripheral Component Interconnect Express (PCIe) interface configured with each of the CPU devices, the data stream; and
aggregating, by the upstream aggregator module, each of the PCIe interfaces to aggregate the data stream.

11. The method of claim 10 wherein the upstream aggregator module is provided on a routing backplane within the rack module.

12. The method of claim 10 further comprising configuring, by one or more PCIe End Point (EP) modules coupled to the downstream aggregator module; the data stream.

13. The method of claim 12 wherein the downstream aggregator module and the one or more PCIe EP modules are integrated on a silicon material.

14. The method of claim 10 wherein the downstream aggregator module and the upstream aggregator module are configured in a 1-to-1 mapping.

15. The method of claim 10 further comprising communicating, by a high-speed link configured between the downstream aggregator module and the upstream aggregator module, the data stream.

16. The method of claim 15 wherein communicating, by the high-speed link, the data stream includes communicating at a transfer rate of greater than 24 Gbps or greater than 36 Gbps.

17. The method of claim 15 further comprising converting, by a pair of optical converters coupled by an optical fiber configured between the upstream aggregator module and the downstream aggregator module, the data stream.

18. A method of operating a data center system having a computer network system configured in a leaf-spine architecture; the computer network system having a router coupled to a network source, a plurality of core switches coupled to the router, a plurality of aggregator switches coupled to each of the plurality of core switches, and a plurality of rack modules having disaggregated Input/Output (I/O) coupled to each of the plurality of aggregator switches, each of the rack modules including an I/O appliance, a plurality of server devices coupled to the I/O appliance, and an upstream aggregator module coupled to the plurality of service devices; the method comprising:
receiving, by a plurality of optical ports of the I/O appliance numbered from 1 to N, a data stream;
routing, by a routing element of the I/O appliance coupled to the plurality of optical ports, the data stream;
configuring, by a Network Interface Controller (NIC) interface of the I/O appliance coupled to the routing element, the data stream;
aggregating, by a downstream aggregator module of the I/O appliance coupled to the NIC interface, the data stream;
wherein the downstream aggregator module is configured as a logical Peripheral Component Interconnect Express (PCIe) switch downstream port;
wherein each of the NIC interfaces is coupled to a top of rack switch and

Assignees

• Inphi Corp

Inventors

• Krishnan Sreenivas
• Saxena Nirmal Raj

Classifications

• H04B10/27

  Arrangements for networking · CPC title

• G06F13/4027

  using bus bridges (G06F13/4022 takes precedence) · CPC title

• H04J14/0267

  Optical signaling or routing · CPC title

• G06F13/4022 Primary

  using switching circuits, e.g. switching matrix, connection or expansion network (G06F13/4009 takes precedence) · CPC title

• H04Q2011/009

  Topology aspects · CPC title