Intermediary duplication to facilitate copy requests in distributed storage systems

Patent metadata

Field	Value
Publication number	US-11023157-B2
Application number	US-201815967284-A
Country	US
Kind code	B2
Filing date	Apr 30, 2018
Priority date	Apr 30, 2018
Publication date	Jun 1, 2021
Grant date	Jun 1, 2021

Abstract

Generally described, aspects of the present application correspond to enabling rapid duplication of data within a data volume hosted on a network storage system. The network storage system can maintain a highly distributed replica of the data volume, designated for duplication of data within the volume and separate from one or more other replicas designated for handling modifications to the data volume. By providing increased parallelization, the highly distributed replica can facilitate rapid duplication of the volume. When a sufficiently large request to duplicate the data volume is received, the system can create additional duplicate portions of the volume to further increase parallelization. For example, a partition of the highly distributed replica may be repeatedly duplicated to create a large number of intermediary duplicate partitions. The intermediary duplicate partitions can then be used to service the duplication request rapidly, due to increased parallelism.

First claim

What is claimed is:

1. A system comprising:
a set of server computing devices implementing a distributed replica of a data volume, the distributed replica including a set of partitions distributed among at least some of the set of server computing devices, wherein the distributed replica is designated for replication of the data volume and is distinct from an additional replica designated for handling modifications to the data volume; and
a coordinator computing device configured with computer-executable instructions to:
receive one or more requests indicating that information within a partition, of the set of partitions, is to be copied to a set of target volumes;
determine that a number of copy operations required to copy the information within the partition to the set of target volumes satisfies or exceeds a threshold value, wherein the threshold value comprises a threshold number of copy operations or a threshold time for completing the number of copy operations utilizing the partition;
duplicate the partition within the set of server computing devices to generate intermediary duplicate partitions;
initiate, in parallel, a set of copy operations copying the information from the intermediary duplicate partitions to the set of target volumes;
determine that a number of outstanding copy operations required to copy the partition falls at or below the threshold value; and
delete one or more of the intermediary duplicate partitions.

2. The system of claim 1 , wherein the one or more requests indicate that the entirety of the volume is to be copied to the set of target volumes, and wherein duplication of the partition to generate intermediary duplicate partitions further comprises duplication of the set of partitions to generate intermediary duplicate sets of partitions.

3. The system of claim 1 , wherein the set of copy operations further comprise a copy operation from the partition to the set of target volumes.

4. The system of claim 1 , wherein the set of copy operations correspond to a first set of copy operations copying the information from the intermediary duplicate partitions to a first subset of the set of target volumes, and wherein the controller computing device is further configured with computer-executable instructions to initiate, in parallel, a second set of copy operations copying the information from the intermediary duplicate partitions to a second subset of the set of target volumes.

5. A computer-implemented method comprising:
implementing a distributed replica of a data volume, the distributed replica including a set of partitions distributed among at least some of a set of server computing devices, wherein the distributed replica is designated for replication of the data volume and is distinct from an additional replica designated for handling modifications to the data volume;
receiving an indication that information within a partition, of the set of partitions, is to be copied to a set of network devices;
duplicating the partition within the set of server computing devices to generate intermediary duplicate partitions;
initiating, in parallel, a set of copy operations copying the information from the intermediary duplicate partitions to the set of network devices;
determining that a number of outstanding copy operations required to copy the partition to the set of network devices falls at or below a threshold level, wherein the threshold level comprises a threshold number of outstanding copy operations or a threshold time for completing the number of outstanding copy operations utilizing the partition; and
deleting one or more of the intermediary duplicate partitions.

6. The computer-implemented method of claim 5 , wherein the indication is received at a first server computing device, of the set of server computing devices, hosting the partition, and wherein duplicating the partitions within the set of server computing devices comprises selecting, at the first server computing device, a second server computing device of the set of server computing devices on which to generate a first intermediary duplicate partition of the intermediary duplicate partitions.

7. The computer-implemented method of claim 6 further comprising, by the first server computing device, instructing the second server computing device to initiate at least a portion of the set of copy operations.

8. The computer-implemented method of claim 7 further comprising, at the second server computing device:
selecting a third server computing device of the set of server computing devices on which to generate a second intermediary duplicate partition of the intermediary duplicate partitions; and
instructing the third server computing device to initiate at least a portion of the set of copy operations.

9. The computer-implemented method of claim 5 , wherein the indication is received at a controller computing device, wherein the partition is hosted at a first server computing device of the set of server computing devices, and wherein duplicating the partitions within the set of server computing devices comprises, at the controller computing device:
selecting a second server computing device of the set of server computing devices to host a first intermediary duplicate partition of the intermediary duplicate partitions; and
transmitting instructions to the first server computing device to create the first intermediary duplicate partition at least partly duplicating the partition to the second server computing device.

10. The computer-implemented method of claim 9 further comprising, at the controller computing device:
receiving an indication that the first intermediary duplicate partition has been created;
selecting a third server computing device of the set of server computing devices to host a second intermediary duplicate partition of the intermediary duplicate partitions; and
transmitting instructions to the second server computing device to create the second intermediary duplicate partition at least partly duplicating the first intermediary duplicate partition to the third server computing device.

11. Non-transitory computer-readable media comprising instructions executable on a system implementing a distributed replica of a data volume, the distributed replica including a set of partitions distributed among at least some of a set of server computing devices, wherein the distributed replica is designated for replication of the data volume and is distinct from an additional replica designated for handling modifications to the data volume, and wherein the instructions are executable by the system to:
receive an indication that information within a partition, of the set of partitions, is to be copied to a set of network devices;
duplicate the partition within the set of server computing devices to generate intermediary duplicate partitions;
initiate, in parallel, a set of copy operations copying the information from the intermediary duplicate partitions to the set of network devices;
determine that a number of outstanding copy operations required to copy the partition to the set of network devices falls at or below a threshold level, wherein the threshold level comprises a threshold number of outstanding copy operations or a threshold time for completing the number of outstanding copy operations utilizing the partition; and
delete the intermediary duplicate partitions.

12. The non-transitory computer-readable media of claim 11 , wherein the instructions are further executable by the system to determine that a number of copy operations required to copy the information within the partition to the set of network devices satisfies or exceeds the threshold value.

Assignees

• Amazon Tech Inc

Inventors

• Greenwood Christopher Magee
• Wei Danny
• Muniswamy-Reddy Kiran-Kumar
• Lin Wells
• Kostic Igor A
• Williams Colin

Classifications

• G06F3/067

  Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS] · CPC title

• G06F3/0652

  Erasing, e.g. deleting, data cleaning, moving of data to a wastebasket · CPC title

• G06F3/061

  Improving I/O performance · CPC title

• G06F3/0644

  Management of space entities, e.g. partitions, extents, pools · CPC title

• G06F3/065 Primary

  Replication mechanisms · CPC title