Shadowing storage gateway

Patent metadata

Field	Value
Publication number	US-2016205187-A1
Application number	US-201615076571-A
Country	US
Kind code	A1
Filing date	Mar 21, 2016
Priority date	Jun 30, 2011
Publication date	Jul 14, 2016
Grant date	—

Abstract

Methods, apparatus, and computer-accessible storage media for shadowing data stored on a local store to a remote store provided by a service provider. A gateway may be configured as a shadowing gateway on a customer network in response to receiving configuration information. The shadowing gateway may receive reads and writes to the local store. The gateway passes the requests to the local store, and also uploads write data indicated by the writes to the service provider to update a snapshot of the local store maintained by the service provider on the remote store. The write data may be buffered to a write log for uploading, and may be uploaded as blocks according to a block storage format used by the service provider. The shadowing process may be transparent to processes on the customer network. The shadowed data may be used to recover data on the local store.

First claim

1 .- 30 . (canceled)

31 . A method, comprising:
receiving, by a storage gateway on a network from one or more processes on the network, read requests and write requests directed to a primary data store at the network, wherein the storage gateway is implemented by one or more computing device on the network comprising one or more hardware processors and memory; for the read requests, passing the read requests to the primary data store at the network; and for the write requests:
passing the write requests to the primary data store to write the primary data store; and
buffering write data indicated by the write requests in a write log with other write data from one or more other write requests; and
uploading data from the write log to a service to update a remote data store in a service provider network.

32 . The method as recited in claim 31 , further comprising initiating, by the storage gateway, a bootstrapping process that uploads at least a portion of the data stored on the local data store to the service provider to make a snapshot of the primary data store consistent with the primary data store.

33 . The method as recited in claim 32 , wherein receiving the read requests and the write requests, passing the read requests and the write to the primary data store, and uploading data to the service are performed by the storage gateway concurrently with the bootstrapping process.

34 . The method as recited in claim 31 , further comprising:
changing the storage gateway to a pass-through mode in which uploading data to the service is suspended for a period; changing the storage gateway to exit the pass-through mode; and subsequent to exiting the pass-through mode, uploading, by the gateway process, at least a portion of the data stored on the primary data store to the service provider to make the snapshot of the primary data store consistent with the primary data store.

35 . The method as recited in claim 31 , wherein said uploading data to the service is performed asynchronously to said passing the read requests and the write requests to the primary data store and to returning responses to the read requests and the write requests from the primary data store to the one or more processes.

36 . The method as recited in claim 31 , wherein the primary data store comprises one or more local storage devices, the method further comprising restoring at least some data on at least one of the one or more local storage devices from a snapshot of the primary data store on the remote data store.

37 . The method as recited in claim 31 , wherein the storage gateway receives the read and write requests from the one or more processes on the network via a block storage protocol.

38 . A device, comprising:
at least one processor; and a memory comprising program instructions, wherein the program instructions are executable by the at least one processor to implement at a network a gateway process configured to:
receive, from one or more processes on the network, read requests and write requests directed to a local data store on the network;
pass the read requests and the write requests to the local data store; and
buffer write data indicated by the write requests in a write log at the network; and
upload data from the write log to a service at a service provider network to update a remote data store at the service provider network.

39 . The device as recited in claim 38 , wherein the gateway process is further configured to upload at least a portion of the data stored on the local data store to the service to make the snapshot of the local data store consistent with the local data store.

40 . The device as recited in claim 38 , wherein the gateway process is further configured to perform said upload asynchronously to said pass the read requests and the write requests to the local data store and to returning responses to the read requests and the write requests from the local data store to the initiating processes.

41 . The device as recited in claim 38 , wherein the gateway process is further configured to initiate a process that registers the gateway process with the service.

42 . The device as recited in claim 38 , wherein the gateway process is further configured to restore at least some data in the local data store from a snapshot of the local data store maintained by the service at the service provider network.

43 . The device as recited in claim 42 , wherein the snapshot of the local data store is maintained on the remote data store by the service in a block format, and wherein, to upload data to the service, the gateway process is further configured to upload blocks of data modified by the write data to the service provider.

44 . The device as recited in claim 38 , wherein said upload is performed according to a data deduplication technique.

45 . A non-transitory computer-accessible storage medium storing program instructions computer-executable to implement a gateway process configured to:
configure as a shadowing gateway on a network according to configuration information received from a service at a service provider network, wherein a shadowing gateway shadows data stored on a local data store to a remote data store at the service provider network; wherein, to shadow data stored on the local data store to the remote data store, the gateway process is configured to:
receive write requests directed to the local data store from one or more processes on the network; and
buffer write data indicated by the write requests in a write log at the network; and
upload data from the write log to the service at the service provider network to update the remote data store at the service provider network.

46 . The non-transitory computer-accessible storage medium as recited in claim 45 , wherein, subsequent to said configuring, the gateway process is configured to upload at least a portion of the data stored on the local data store to the service to make a snapshot of the local data store consistent with the local data store.

47 . The non-transitory computer-accessible storage medium as recited in claim 45 , wherein the gateway process is further configured to pass the write requests to the local data store, and wherein the gateway process is configured to perform said upload asynchronously to said passing the write requests to the local data store.

48 . The non-transitory computer-accessible storage medium as recited in claim 45 , wherein, to upload data to the service provider, the gateway process is further configured to upload the data according to a Web service interface of the service.

49 . The non-transitory computer-accessible storage medium as recited in claim 45 , wherein the gateway process is configured to receive the read and write requests from the one or more processes on the network via a block storage protocol.

50 . The non-transitory computer-accessible storage medium as recited in claim 45 , wherein the gateway process is configured to perform said upload according to a data deduplication technique.

Assignees

• Amazon Tech Inc

Inventors

• Sorenson Iii James Christopher
• Lin Yun
• Salyers David Carl
• Khetrapal Ankur

Classifications

• G06F11/1464

  for networked environments · CPC title

• G06F16/1724

  Details of de-fragmentation performed by the file system (saving storage space on storage systems G06F3/0608; management of blocks in storage devices G06F3/064) · CPC title

• H04L67/1097

  for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS] · CPC title

• H04L67/02

  based on web technology, e.g. hypertext transfer protocol [HTTP] · CPC title

• H04L67/1095 Primary

  Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes · CPC title