Retention-based data management in a network-based data store

Patent metadata

Field	Value
Publication number	US-9912752-B1
Application number	US-201514754570-A
Country	US
Kind code	B1
Filing date	Jun 29, 2015
Priority date	Jun 29, 2015
Publication date	Mar 6, 2018
Grant date	Mar 6, 2018

Abstract

A network-based data store may implement retention-based management techniques for data stored at the network-based data store. When data is received for storage at the network-based data store, a retention time for the data may be determined. Storage locations at persistent storage devices of the network-based data store may be selected according to the retention time. The data may then be placed at the storage locations. When a request to delete data is received, retention times of co-located data may be evaluated to determine whether the deletion may be delayed. Delayed deletions may allow the data to be subsequently deleted with at least some of the co-located data. Repair operations to maintain the data according to a durability policy may be modified according to retention time for a data suffering a loss of redundancy.

First claim

What is claimed is:

1. A system, comprising:
a plurality of persistent storage devices that store data for a network-based data store;
a plurality of compute nodes that implement a control plane for the network-based data store;
a network-based interface for the network-based data store;
the control plane, configured to:
receive a request to store a data object via the network-based interface at the network-based data store, wherein the request indicates a data retention time for the data object;
identify a plurality of candidate storage locations at the persistent storage devices to place the data object, wherein one of the plurality of candidate storage locations at a persistent storage device of the persistent storage devices is identified among other locations at the persistent storage device according to the data retention time;
place the data object at one or more of the candidate storage locations.

2. The system of claim 1 , wherein the control plane is further configured to:
receive another request to store a different data object via the network-based interface at the network-based data store;
apply one or more retention rules to determine a retention time for the other data object;
identify a plurality of other candidate storage locations at the persistent storage devices to place the other data object according to the data retention time for the other data object; and
place the other data object at one or more of the other candidate storage locations.

3. The system of claim 1 , wherein the control plane is further configured to:
receive a request to delete the data object from the network-based data store;
identify respective retention times for one or more other data objects stored at the one or more candidate storage locations storing the data object; and
based, at least in part, on the respective retention times for the one or more other data objects, mark the data for subsequent deletion in order to delay deletion of the data object until deleting at least one of the other data objects.

4. The system of claim 1 , wherein the plurality of persistent storage devices are shingled magnetic recording devices, wherein the network-based data store is an object storage service, and wherein the data is stored a client for a client of the object storage service and associated with a particular customer account.

5. A method, comprising:
performing, by one or more computing devices:
receiving data to be stored in a network-based data store;
determining a retention time for maintaining the data in the network-based data store;
selecting one or more storage locations at one or more persistent storage devices implemented as part of the network-based data store to place the data, wherein selecting one of the one or more storage locations at a persistent storage device of the one or more persistent storage devices comprises selecting among locations at the persistent storage device according to the data retention time; and
placing the data at the one or more storage locations.

6. The method of claim 5 , wherein determining the retention time for maintaining the data in the network-based data store comprises evaluating a request from a client to store the data that indicates a retention time for the data.

7. The method of claim 5 , wherein determining the retention time for maintaining the data in the network-based data store comprises applying one or more retention rules to determine the retention time.

8. The method of claim 5 , further comprising:
receiving a request to delete the data from the network-based data store;
identifying one or more respective retention times for other data stored at the one or more storage locations storing the data; and
based, at least in part, on the one or more respective retention times for the other data, delaying deletion of the data such that the data is deleted with at least some of the other data.

9. The method of claim 5 , wherein the one or more persistent storage devices are one or more shingled magnetic recording devices.

10. The method of claim 9 , wherein selecting the one or more storage locations at the one or more persistent storage devices comprises identifying a portion of a region of the one or more shingled magnetic recording devices that overlaps other data with a longer retention time.

11. The method of claim 5 , wherein the data is stored as a plurality of data replicas at the one or more storage locations in the network-based data store in order to satisfy a durability policy for the data, and wherein the method further comprises:
detecting a failure of one or more of the replicas of the data;
evaluating the retention time of the data with respect to the durability policy; and
based, at least in part, on the evaluation, determining that remaining live replicas of the data satisfy the durability policy for the data.

12. The method of claim 5 ,
wherein a request is received from a client to store the data that indicates that the data is sent incrementally;
wherein receiving the data comprises receiving a plurality of chunks of the data via a streaming connection between the network-based data store and the client; and
wherein the determined retention time is a same retention time for the plurality of chunks of the data;
wherein selecting the one or more storage locations is based, at least in part, on determining space for storing one or more subsequent chunks of the data in addition to a first chunk of the data; and
wherein placing the data comprises appending the plurality of data chunks to the one or more storage locations as the data chunks are received.

13. The method of claim 5 , wherein the network-based data store is a storage service implemented as part of a provider network, and wherein the data is stored for a client of the storage service and associated with a particular customer account of the provider network.

14. A non-transitory, computer-readable storage medium, storing program instructions that when executed by one or more computing devices cause the one or more computing devices to implement:
receiving a request to store data in a network-based data store;
determining a retention time for maintaining the data in the network-based data store;
identifying a plurality of candidate storage locations at a plurality of persistent storage devices implemented as part of the network-based data store to place the data according to the data retention time, comprising determining a retention time for other data stored at one or more of the candidate storage locations that would be overlapped by the data upon placement of the data at the one or more storage locations, wherein respective ones of the persistent storage devices that implement the one or more candidate storage locations are shingled magnetic recording devices; and
placing the data at one or more of the candidate storage locations.

15. The non-transitory, computer-readable storage medium of claim 14 , wherein the program instructions cause the one or more computing devices to further implement:
receiving a request to store different data at the network-based data store, wherein the request indicates a data retention time for the different data;
identifying a plurality of other candidate storage locations at the persistent storage devices to place the different data according to the data retention time for the different data; and
placing the different data at one or more of the other candidate storage locations.

16. The non-transitory, computer-readable storage medium of claim 14 , wherein the program instructions cause the o

Assignees

• Amazon Tech Inc

Inventors

• Davis Timothy James
• Patel Rajesh Shanker
• Marshall Bradley Eugene
• Collins Jonathan Robert

Classifications

• H04L67/1097 Primary

  for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS] · CPC title

• H04L67/18

  Electricity · mapped topic

• G06F16/2365

  Ensuring data consistency and integrity · CPC title