Asynchronous movement of in-line metadata for cached volumes at storage gateways

Patent metadata

Field	Value
Publication number	US-9588895-B2
Application number	US-201615050408-A
Country	US
Kind code	B2
Filing date	Feb 22, 2016
Priority date	Oct 31, 2012
Publication date	Mar 7, 2017
Grant date	Mar 7, 2017

Abstract

Methods and apparatus for supporting cached volumes at storage gateways are disclosed. A storage gateway appliance is configured to cache at least a portion of a storage object of a remote storage service at local storage devices. In response to a client's write request, directed to at least a portion of a data chunk of the storage object, the appliance stores a data modification indicated in the write request at a storage device, and asynchronously uploads the modification to the storage service. In response to a client's read request, directed to a different portion of the data chunk, the appliance downloads the requested data from the storage service to the storage device, and provides the requested data to the client.

First claim

What is claimed is:

1. A system, comprising:
one or more computing devices comprising one or more hardware processors and one or more memories storing program instructions that, when executed by at least one of the one or more hardware processors, configure the at least one of the one or more computing devices to:
cache, by a storage appliance, at least a portion of one or more data chunks of a storage object;
store, by the storage appliance, inline metadata for the one or more data chunks;
store, by the storage appliance, contiguous metadata for the one more data chunks;
in response to transfer of a new data block from a remote storage service for a particular data chunk of the one or more data chunks, synchronously update the inline metadata without synchronously updating the contiguous metadata;
perform an asynchronous update to the contiguous metadata with respect to the transfer of the new data block and the synchronous update of the inline metadata, wherein the asynchronous update causes one or more changes to the contiguous metadata stored on the storage appliance; and
access the contiguous metadata to determine a state of at least a portion of the particular data chunk without reading all the inline metadata for the particular data chunk.

2. The system recited in claim 1 , wherein the contiguous metadata indicates offset and state information for the one or more data chunks of the storage object.

3. The system recited in claim 1 , wherein the one or more computing devices are further configured to determine, using the inline metadata associated with the one or more data chunks, whether an I/O request directed to at least a portion of the one or more data chunks results in a cache miss.

4. The system recited in claim 3 ,
wherein the one or more data chunks and a corresponding inline metadata for the one or more data chunks are stored together in an alternating sequence of cached data blocks of the particular data chunk and inline metadata sections corresponding to the cached data blocks, and
wherein the one or more computing devices are further configured to update a particular portion of the contiguous metadata based at least in part on a modification of at least one element of inline metadata, wherein said updating the particular portion of the contiguous metadata is performed asynchronously with respect to the modification of the at least one element of inline metadata.

5. The system recited in claim 3 , wherein to modify the at least one element of inline metadata, the one or more computing devices are further configured to:
identify a first inline metadata section adjacent to a particular data block within the particular data chunk, and a different inline metadata section adjacent to the particular data block within the particular data chunk; and
modify the first inline metadata section and the different inline metadata section prior to providing a response to the I/O request.

6. The system recited in claim 5 , wherein to modify the at least one elements of inline metadata, the one or more computing devices are further configured to identify an offset at which the at least one element of inline metadata is stored, based at least in part on an offset of the particular data block within the particular data chunk.

7. A method, comprising:
caching, by a storage appliance, at least a portion of one or more data chunks of a storage object;
storing, by the storage appliance, inline metadata for the one or more data chunks;
storing, by the storage appliance, contiguous metadata for the one more data chunks;
subsequent to transfer of a new data block from a remote storage service for a particular data chunk of the one or more data chunks, synchronously updating the inline metadata without synchronously updating the contiguous metadata;
asynchronously updating the contiguous metadata with respect to the transfer of the new data block and the synchronous updating of the inline metadata, wherein asynchronously updating causes one or more changes to the contiguous metadata stored on the storage appliance; and
accessing the contiguous metadata to determine a state of at least a portion of the particular data chunk without reading all the inline metadata for the particular data chunk.

8. The method recited in claim 7 , wherein the contiguous metadata indicates offset and state information for the one or more data chunks of the storage object.

9. The method recited in claim 7 , further comprising:
receiving a read request directed to a particular data block of the particular data chunk;
downloading at least a portion of the particular data chunk, comprising the particular data block, from the remote storage service;
determining whether the at least a portion of the particular data chunk is to be merged with one or more cached data blocks;
subsequent to determining that a merge is to be performed, merging the at least a portion of the particular data chunk with the one or more cached data blocks prior to storing the at least a portion of the particular data chunk; and
extracting the particular data block from the at least a portion of the particular data chunk and providing the particular data block in response to the read request.

10. The method recited in claim 7 , further comprising:
receiving an I/O request comprising a modification directed to a particular data block of the particular data chunk;
modifying the particular data block in accordance with the I/O request;
adding an entry to an upload buffer configured for asynchronous uploads of modified data to the remote storage service, wherein the entry indicates modifications performed on the particular data block in response to the I/O request; and
verifying that modifications indicated in the entry have been uploaded to the remote storage service prior to replacing the particular data block.

11. The method recited in claim 10 ,
wherein the one or more data chunks and a corresponding inline metadata for the one or more data chunks are stored together in an alternating sequence of cached data blocks of the particular data chunk and inline metadata sections corresponding to the cached data blocks, and
wherein the method further comprises updating a particular portion of the contiguous metadata based at least in part on modifying at least one element of inline metadata, wherein said updating the particular portion of the contiguous metadata is performed asynchronously with respect to the modification of the at least one element of inline metadata.

12. The method recited in claim 11 , wherein modifying the at least one element of inline metadata includes identifying an offset at which the at least a portion of inline metadata is stored, based at least in part on an offset of the particular data block within the particular data chunk.

13. The method recited in claim 12 , wherein modifying the at least one element of inline metadata includes:
identifying a first inline metadata section adjacent to the particular data block, and a different inline metadata section adjacent to the particular data block; and
modifying the first inline metadata section and the different inline metadata section prior to providing a response to the I/O request.

14. A non-transitory computer-readable storage medium storing program instructions that when executed by one or more computer processors:
cache at least a portion of one or more data chunks of a storage object;
store inline metadata for the one or more data chunks;
store contiguous metadata for the one more data chunks;
subsequent to transfer of a new data block from a remote storage service for a particular data c

Assignees

• Amazon Tech Inc

Inventors

• Salyers David Carl
• Vincent Pradeep
• Khetrapal Ankur
• Patiejunas Kestutis

Classifications

• G06F11/1464

  for networked environments · CPC title

• G06F16/27

  Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor · CPC title

• G06F12/0893

  Caches characterised by their organisation or structure · CPC title

• G06F2212/468

  The specific object being partially cached · CPC title

• G06F3/0685

  Hybrid storage combining heterogeneous device types, e.g. hierarchical storage, hybrid arrays · CPC title