Distributed in-memory buffer cache system using buffer cache nodes

Patent metadata

Field	Value
Publication number	US-9880933-B1
Application number	US-201314085668-A
Country	US
Kind code	B1
Filing date	Nov 20, 2013
Priority date	Nov 20, 2013
Publication date	Jan 30, 2018
Grant date	Jan 30, 2018

Abstract

A separate distributed buffer cache system may be implemented for a storage client of a distributed storage system. Storage I/O requests may be sent from a storage client to one or more buffer cache nodes in a distributed buffer cache system that maintain portions of an in-memory buffer cache to which the requests pertain. The distributed buffer cache system may send the write requests on to the distributed storage system to be completed, and in response to receiving acknowledgements from the storage system, sending a completion acknowledgement back to the storage client. Buffer cache nodes may update buffer cache entries for received requests such that they are not available for reads until complete at the distributed storage system. For read requests where the buffer cache entries at the buffer cache node are invalid, valid data may be obtained from the distributed storage system and sent to the storage client.

First claim

What is claimed is:

1. A system, comprising:
a plurality of storage nodes, each comprising one or more respective processors, implementing a distributed storage system that maintains data for at least one storage client;
a plurality of buffer cache nodes, each comprising one or more respective processors, implementing a distributed buffer cache system that maintains an in-memory, write-through buffer cache for access operations directed toward the data maintained for the at least one storage client at the distributed storage system, wherein the buffer cache nodes are configured to access the data stored for the at least one storage client at the distributed storage system and maintain different portions of the buffer cache;
at least one compute node implementing the at least one storage client, configured to:
determine a storage I/O (Input/Output) write request to be performed at the distributed storage system;
identify one or more buffer cache nodes of the plurality of buffer cache nodes to send the storage I/O write request, wherein the identified one or more buffer cache nodes maintains at least a portion of the write-through buffer cache to which the write request pertains and buffer cache metadata reflecting a state of the portion of the buffer cache in a respective system memory of the one or more buffer cache nodes;
send the storage I/O write request to the one or more buffer cache nodes; and
receive from at least one of the one or more buffer cache nodes an acknowledgment that the storage I/O write is complete at the distributed storage system.

2. The system of claim 1 , wherein the at least one node implementing the storage client is further configured to:
determine a storage I/O read request for specified data of the data maintained for the at least one storage client at the distributed storage system;
determine one or more buffer cache nodes of the plurality of buffer cache nodes to send the storage I/O read request, wherein the determined one or more buffer cache nodes maintain at least a portion of the write-through buffer cache to which the storage I/O read request pertains;
send the storage I/O read request to the determined one or more buffer cache nodes; and
receive from at least one of the one or more buffer cache nodes the specified data in order to service the storage I/O read request.

3. The system of claim 2 , wherein at least one of the plurality of buffer cache nodes is configured to:
receive the storage I/O read request for the specified data from the at least one compute node implementing the at least one storage client;
determine that one or more buffer cache entries for the specified data are valid; and
respond to the at least one storage client with the specified data obtained from the one or more buffer cache entries.

4. The system of claim 2 , wherein at least one of the plurality of buffer cache nodes is configured to:
receive the storage I/O read request for the specified data from the at least one compute node implementing the at least one storage client;
determine that one or more buffer cache entries for the specified data are invalid;
in response to determining that the one or more buffer cache entries for the specified data are invalid:
request the specified data from one or more storage nodes maintaining the specified data in the distributed storage system;
receive the specified data from the one or more storage nodes;
update the one or more buffer cache entries with the received specified data such that the one or more buffer cache entries are available for reads; and
respond to the at least one storage client with the received specified data.

5. The system of claim 1 , wherein the storage client is a network-based database service, wherein the distributed storage system is a log-structured data store, and wherein the write request is a log record indicating an update to the data maintained at the log-structured data store for the network-based database service.

6. A method, comprising:
performing, by a plurality of computing devices:
maintaining an in-memory buffer cache among one or more buffer cache nodes, each comprising one or more respective processors, in a distributed buffer cache system for access operations received from at least one storage client and directed toward data stored at a distributed storage system for the storage client, each buffer cache node maintaining a plurality of buffer cache entries of a different portion of buffer cache and buffer cache metadata reflecting a respective state of buffer cache entries;
receiving, at a buffer cache node of the one or more buffer cache nodes, a storage I/O (input/output) write request to be performed at the distributed storage system from the at least one storage client;
in response to receiving the storage I/O write request:
sending the storage I/O write request to the distributed storage system to be performed;
based, at least in part, on one or more acknowledgments received for the storage I/O write request from the distributed storage system, determining that the storage I/O write request is complete at the distributed storage system;
updating one or more buffer cache entries and respective buffer cache metadata maintained at the buffer cache node according to the storage I/O write request, wherein the updated one or more buffer cache entries are not available for reads until the storage I/O write request is determined to be complete at the distributed storage system; and
acknowledging the completion of the write request to the at least one storage client.

7. The method of claim 6 , wherein the distributed storage system comprises a plurality of storage nodes, wherein the one or more acknowledgements are received from one or more different storage nodes of the plurality of storage nodes, and wherein said determining that the storage I/O write request is complete at the distributed storage system, comprises determining that the one or more acknowledgments satisfy a write quorum requirement for the storage I/O write request.

8. The method of claim 6 , further comprising:
identifying the one or more buffer cache entries as write-in-progress such that subsequent storage I/O read requests for data maintained in the one or more buffer cache entries are blocked;
in response to determining that the storage I/O write request is complete at the distributed storage system, identifying the one or more buffer cache entries as available for servicing subsequent storage I/O read requests.

9. The method of claim 6 , further comprising:
receiving, at the buffer cache node, a storage I/O read request from the at least one storage client for specified data of the data maintained at the distributed storage system;
in response to receiving the storage I/O read request:
determining that one or more buffer cache entries for the specified data are valid; and
responding to the at least one storage client with the specified data obtained from the one or more buffer cache entries.

10. The method of claim 6 , further comprising:
receiving, at the buffer cache node, a storage I/O read request for specified data of the data maintained at the distributed storage system;
in response to receiving the storage I/O read request:
determining that one or more buffer cache entries for the specified data are invalid;
requesting the specified data from the distributed storage system;
receiving the specified data from the distributed storage system;
updating the one or more buffer cache entries corresponding to the received specified data such that the one or more buffer cache entries are available for reads; and
responding to the at least one storage client with the specified data obtained from

Assignees

• Amazon Tech Inc

Inventors

• Gupta Anurag Windlass
• Allen Matthew David

Classifications

• G06F3/067

  Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS] · CPC title

• G06F17/30224

  Physics · mapped topic

• G06F12/0868 Primary

  Data transfer between cache memory and other subsystems, e.g. storage devices or host systems · CPC title

• G06F12/0815 Primary

  Cache consistency protocols · CPC title

• G06F2211/1028

  Distributed, i.e. distributed RAID systems with parity · CPC title