Scale-out high bandwidth memory system

Patent metadata

Field	Value
Publication number	US-12032497-B2
Application number	US-202117469769-A
Country	US
Kind code	B2
Filing date	Sep 8, 2021
Priority date	Sep 20, 2018
Publication date	Jul 9, 2024
Grant date	Jul 9, 2024

Abstract

A high bandwidth memory (HBM) system includes a first HBM+ card. The first HBM+ card includes a plurality of HBM+ cubes. Each HBM+ cube has a logic die and a memory die. The first HBM+ card also includes a HBM+ card controller coupled to each of the plurality of HBM+ cubes and configured to interface with a host, a pin connection configured to connect to the host, and a fabric connection configured to connect to at least one HBM+ card.

First claim

What is claimed is:

1. A memory system comprising:
a first memory card, wherein the first memory card comprises:
a memory device, wherein the memory device comprises a first logic die and a memory die;
a controller connected to the memory device and configured to communicate with a host;
a first connection configured to connect to the host; and
a first fabric connection; and

a second memory card, wherein the second memory card comprises:
a second logic die for performing a computation and generating a first output;
a second fabric connection,

wherein the second memory card is configured to transmit the first output to the first memory card via the second fabric connection that connects to the first memory card via the first fabric connection, wherein the first memory card is configured to generate a second output, and provide the second output to the host via the first connection.

2. The memory system of claim 1 , wherein the first logic die or the second logic die comprises an accelerator logic configured to:
receive instructions from the controller;
input vectors to a computational component;
execute a mathematical operation; and
return an output to an accumulator.

3. The memory system of claim 1 , wherein the first logic die or the second Iodic die comprises an accelerator, wherein the accelerator comprises:
a control component;
a buffer;
an instruction decoder; and
a general matrix multiply (GEMM) component.

4. The memory system of claim 3 , wherein the control component is configured to operate as at least one of a routing controller, a high bandwidth memory controller, a direct memory access (DMA) engine, a power controller, or a multiple model adaptive controller (MMAC) scheduler.

5. The memory system of claim 3 , wherein the GEMM component comprises one or more of:
a first multiple model adaptive controller (MMAC);
a second MMAC; or
a multiplexer configured to route first data to the second MMAC and route second data to first MMAC.

6. The memory system of claim 1 , wherein the memory die comprises at least one volatile memory component.

7. The memory system of claim 1 , wherein the first logic die is stacked on top of the memory die.

8. The memory system of claim 1 , wherein the memory device is configured to send or receive data to another memory device in the second memory card using at least one of a buffer-based communication link or peer-to-peer communication link.

9. The memory system of claim 1 , wherein the host provides one or more third connections, wherein a number of the third connections is fewer than a number of memory cards included in the system.

10. The memory system of claim 1 , wherein instructions from the host is configured to be received by the second memory card via a second connection, and transmitted to the first memory card via the second fabric connection.

11. The memory system of claim 10 , wherein the first logic die is configured to perform a computation based on the instructions.

12. A memory system comprising:
a first memory card, wherein the first memory card comprises:
a first memory device, wherein the first memory device comprises a first logic die and a first memory die;
a first controller connected to the first memory device and configured to communicate with a host;
a first connection configured to connect to the host; and
a first fabric connection; and

a second memory card, wherein the second memory card comprises:
a second logic die for performing a computation and generating a first output;
a second fabric connection,

wherein the second memory card is configured to transmit the first output to the first memory card via the second fabric connection that connects to the first memory card via the first fabric connection, wherein the first memory card is configured to generate a second output, and provide the second output to the host via the first connection.

13. The memory system of claim 12 , further comprising:
a third memory card having a third connection configured to connect to the host and a third fabric connection configured to connect to the second memory card; and
a fourth memory card having a fourth connection configured to connect to the host and a fourth fabric connection configured to connect to the second memory card.

14. The memory system of claim 13 , wherein:
the first fabric connection is connected to the second fabric connection, the third fabric connection, and the fourth fabric connection;
the second fabric connection is connected to the first fabric connection, the third fabric connection, and the fourth fabric connection;
the third fabric connection is connected to the first fabric connection, the second fabric connection, and the fourth fabric connection; and
the fourth fabric connection is connected to the first fabric connection, the second fabric connection, and the third fabric connection.

15. The memory system of claim 14 , wherein the third connection and the fourth connection are connected to the host.

16. The memory system of claim 12 , wherein the host provides one or more third connections, wherein a number of the third connections is fewer than a number of memory cards included in the system.

17. The memory system of claim 12 , wherein the computation is based on instructions from the host.

18. The memory system of claim 12 , wherein the first logic die is configured to perform a data computation based on instructions from the host.

19. A memory system comprising:
a first memory card, wherein the first memory card comprises:
a memory device, wherein the memory device is configured to send and receive data to another memory device using at least one of a buffer-based or peer-to-peer communication link, the memory device comprising a memory and a first accelerator;
a controller coupled to the memory device and configured to interface with a host;
a first connection configured to connect to the host; and
a first fabric connection; and

a second memory card, wherein the second memory card comprises:
a second accelerator for performing a computation and generating a first output; and
a second fabric connection,
wherein the second memory card is configured to transmit the first output to the first memory card via the second fabric connection that connects to the first memory card via the first fabric connection, wherein the first memory card is configured to generate a second output, and provide the second output to the host via the first connection.

20. The memory system of claim 19 , wherein the first memory card is configured to operate in accordance with computer program instructions for executing operations on the first accelerator and controlling communication with the memory device.

21. The memory system of claim 1 , wherein the second memory card further comprises:
a second memory device, wherein the second memory device comprises the second logic die and a second memory die; and
a second controller connected to the second memory device and configured to communicate with the host.

22. The memory system of claim 1 , wherein the first output generated by the second memory card is in response to a command from the host.

Assignees

• Samsung Electronics Co Ltd

Inventors

• Malladi Krishna T
• Zheng Hongzhong
• Niu Dimin
• Gu Peng

Classifications

• G06F2212/1016

  Performance improvement · CPC title

• G06F13/1668

  Details of memory controller · CPC title

• G06F13/28

  using burst mode transfer, e.g. direct memory access {DMA}, cycle steal (G06F13/32 takes precedence) · CPC title

• G06F9/30036

  Instructions to perform operations on packed data, e.g. vector, tile or matrix operations · CPC title

• G06F7/5443

  Sum of products (for applications thereof, see the relevant places, e.g. G06F17/10, H03H17/00) · CPC title