Profile-guided data preloading for virtualized resources

Patent metadata

Field	Value
Publication number	US-9563385-B1
Application number	US-201314028186-A
Country	US
Kind code	B1
Filing date	Sep 16, 2013
Priority date	Sep 16, 2013
Publication date	Feb 7, 2017
Grant date	Feb 7, 2017

Abstract

Methods and apparatus for profile-guided preloading for virtualized resources are described. A block-level storage volume whose contents are to be populated via data transfers from a repository service is programmatically attached to a compute instance. An indication of data transfers from the repository to a block storage service implementing the volume is obtained, corresponding to a particular phase of program execution at the compute instance. A storage profile is generated, based at least in part on the indication of data transfers. The storage profile is subsequently used to pre-load data from the repository service on behalf of other compute instances.

First claim

What is claimed is:

1. A system, comprising:
one or more computing devices configured to:
implement a plurality of compute instances at a virtualized compute service of a provider network, wherein a block-level storage volume implemented at a block storage service of the provider network is programmatically attached for block-level network access by a particular compute instance of the plurality of compute instances, and wherein, in response to one or more block-level I/O requests to the block-level storage volume from the particular compute instance, data is transferred from an object at a storage repository service of the provider network to populate the block-level storage volume prior to at least a portion of the data then being provided from the block-level storage volume to the particular compute instance in response to the one or more block-level I/O requests;
obtain, corresponding to a particular phase of program execution at the particular compute instance, an indication of one or more data transfers from the object at the storage repository service to the block storage service;
generate, based at least in part on the indication of the one or more data transfers, a storage access profile associated with the particular phase at the particular compute instance;
store the storage access profile;
in response to a determination that the particular phase of program execution is to occur at a different compute instance of the provider network,
identify at least a portion of an object indicated by the storage access profile to be pre-loaded to populate a different block-level storage volume attached to the different compute instance, wherein the storage access profile generated based on the particular phase of program execution at the particular compute instance is used to identify the portion of the object to be pre-loaded to the different block-level storage volume attached to the different compute instance; and
transfer the at least a portion of the object to the different block-level storage volume.

2. The system as recited in claim 1 , wherein the storage repository service comprises a key-value store configured to store unstructured data objects accessible via a web-services interface.

3. The system as recited in claim 1 , wherein the one or more computing devices are further configured to:
select the particular phase for profile generation based at least in part on one or more of: (a) a determination that the particular phase comprises a sequence of I/O requests expected to be repeated, with a probability above a threshold level, by one or more other compute instances, (b) a determination of performance costs associated with the one or more data transfers from the object at the storage repository service, or (c) a profiling request.

4. The system as recited in claim 1 , wherein the object indicated by the storage access profile comprises a volume snapshot generated in response to a client request.

5. The system as recited in claim 1 , wherein the one or more computing devices are further configured to:
in response to a determination that an I/O request from the different compute instance during the particular phase of program execution cannot be satisfied from the pre-loaded portion of the object,
store a record of the determination that the I/O request could not be satisfied; and
modify the storage access profile based at least in part on the record.

6. A method, comprising:
performing, by one or more computing devices:
programmatically attaching a block-level storage volume implemented at a block storage service of a provider network to a particular compute instance of a virtualized compute service, wherein, in response to one or more block-level I/O requests to the block-level storage volume from the particular compute instance, the block storage service is configured to transfer data from an object at a storage repository service of the provider network to populate the block-level storage volume;
obtaining, corresponding to a particular phase of program execution at the particular compute instance, an indication of one or more data transfers from the object at the storage repository service to the block storage service;
generating, based at least in part on the indication of the one or more data transfers, a storage access profile associated with the particular phase at the particular compute instance, to be used to pre-load data from the object at the storage repository service on behalf of one or more other compute instances different from the particular compute instance; and
storing the storage access profile.

7. The method as recited in claim 6 , further comprising:
determining that a phase of program execution related to the particular phase is to occur at a different compute instance of the provider network;
identifying at least a portion of an object indicated by the storage access profile to be pre-loaded to populate a different block-level storage volume attached to the different compute instance; and
transferring the at least a portion of the object indicated by the storage access profile to populate the different block-level storage volume.

8. The method as recited in claim 6 , further comprising performing, by the one or more computing devices:
implementing, at the storage repository service, a web services programmatic interface to provide access to the object.

9. The method as recited in claim 6 , wherein the particular phase comprises a boot sequence of the particular compute instance.

10. The method as recited in claim 6 , wherein the particular phase comprises a phase of execution of an application program.

11. The method as recited in claim 10 , wherein the application comprises a database management system.

12. The method as recited in claim 6 , wherein the object indicated by the storage access profile comprises a volume snapshot generated in response to a client request.

13. The method as recited in claim 6 , further comprising performing, by the one or more computing devices:
storing a record of a determination that an I/O request generated during the particular phase could not be satisfied by data pre-loaded in accordance with the storage access profile; and
modifying the storage access profile based at least in part on the record.

14. The method as recited in claim 6 , further comprising performing, by the one or more computing devices:
pre-loading at least a portion of a block-level storage volume at a local storage device of a virtualization host at which a different compute instance is implemented, wherein said pre-loading comprises transferring at least a portion of an object from the storage repository service to the local storage device in accordance with the storage access profile.

15. The method as recited in claim 6 , further comprising performing, by the one or more computing devices:
monitoring one or more resource consumption metrics at the provider network;
identifying, from a candidate pre-load data set indicated in the storage access profile, in accordance with the one or more resource consumption metrics, a particular data unit to be transferred from the storage repository service, and a different data unit not to be transferred from the storage repository service, and
transferring the particular data unit from the storage repository service to the block storage service.

16. A non-transitory computer-accessible storage medium storing program instructions that when executed on one or more processors:
obtain, from a collection of storage access profiles generated based at least in part on monitoring, during a ph

Assignees

• Amazon Tech Inc

Inventors

• Kowalski Marcin Piotr
• Richardson David R
• Greenfield James Alfred Gordon
• Van Der Merwe Jacobus Johannes Nicolaas
• Eddey Matthew James
• De Kadt Christopher Richard Jacques
• Desantis Peter Nicholas

Classifications

• G06F3/0673

  Single storage device · CPC title

• G06F3/0665 Primary

  at area level, e.g. provisioning of virtual or logical volumes · CPC title

• G06F3/0604

  Improving or facilitating administration, e.g. storage management · CPC title

• G06F3/065

  Replication mechanisms · CPC title

• G06F3/067

  Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS] · CPC title