Archiving data in cloud/object storage using local metadata staging

Patent metadata

Field	Value
Publication number	US-10853312-B2
Application number	US-201816035439-A
Country	US
Kind code	B2
Filing date	Jul 13, 2018
Priority date	Jul 13, 2018
Publication date	Dec 1, 2020
Grant date	Dec 1, 2020

Abstract

Techniques for archiving data in cloud/object storage using local metadata staging are provided. In one set of embodiments, a computer system residing at an on-premises site comprising on-premises storage can receive a snapshot of a dataset to be archived. The computer system can package data in the snapshot into one or more fixed-size data chunks and upload the one or more fixed-size data chunks to cloud/object storage. Further, concurrently with the packaging and the uploading, the computer system can stage metadata for the snapshot in the on-premises storage. Then, upon uploading all of the data of the snapshot, the computer system can upload the metadata staged in the on-premises storage to the cloud/object storage.

First claim

What is claimed is:

1. A method for archiving data in cloud/object storage using local metadata staging, the method comprising:
receiving, by a computer system residing at an on-premises site comprising on-premises storage, a snapshot of a dataset to be archived, the snapshot including data of the dataset;
packaging, by the computer system, the data included in the snapshot into one or more fixed-size data chunks and uploading the one or more fixed-size data chunks to the cloud/object storage;
concurrently with the packaging and the uploading, staging, by the computer system, metadata for the snapshot in the on-premises storage; and
upon uploading all of the one or more fixed-size data chunks, uploading, by the computer system, the metadata staged in the on-premises storage to the cloud/object storage.

2. The method of claim 1 wherein staging the metadata for the snapshot in the on-premises storage comprises:
creating and updating, in the on-premises storage, a B+ tree for the snapshot comprising internal nodes and leaf nodes, each leaf node representing a data block of the snapshot and including a pointer to a cloud physical block address (CPBA) in the cloud/object storage where the data block has been or will be uploaded.

3. The method of claim 1 wherein uploading the metadata comprises:
packaging the metadata into one or more fixed-size metadata chunks and uploading the one or more fixed-size metadata chunks to the cloud/object storage.

4. The method of claim 1 wherein a predefined range of local physical block addresses (LPBAs) is reserved for snapshot metadata for the dataset in the on-premises storage, and wherein an identical predefined range of CPBAs is reserved for snapshot metadata for the dataset in the cloud/object storage.

5. The method of claim 4 wherein staging the metadata for the snapshot in the on-premises storage comprises:
allocating blocks for the metadata sequentially from the predefined range of LPBAs.

6. The method of claim 5 wherein uploading the metadata staged in the on-premises storage to the cloud/object storage comprises:
packaging the metadata into one or more fixed-size metadata chunks having sequential chunk identifiers from the predefined range of CPBAs and uploading the one or more fixed-size metadata chunks with the sequential chunk identifiers to the cloud/object storage, wherein the CPBA of each block of the metadata in the cloud/object storage maps directly to the LPBA of said each block of the metadata in the on-premises storage.

7. The method of claim 1 wherein staging the metadata for the snapshot in the on-premises storage comprises:
allocating blocks for the metadata randomly from a local physical block address space on the on-premises storage.

8. A non-transitory computer readable storage medium having stored thereon program code executable by a computer system residing at an on-premises site comprising on-premises storage, the program code embodying a method for archiving data in cloud/object storage using local metadata staging, the method comprising:
receiving a snapshot of a dataset to be archived, the snapshot including data of the dataset;
packaging the data included in the snapshot into one or more fixed-size data chunks and uploading the one or more fixed-size data chunks to the cloud/object storage;
concurrently with the packaging and the uploading, staging metadata for the snapshot in the on-premises storage; and
upon uploading all of the data of the snapshot one or more fixed-size data chunks, uploading the metadata staged in the on-premises storage to the cloud/object storage.

9. The non-transitory computer readable storage medium of claim 8 wherein staging the metadata for the snapshot in the on-premises storage comprises:
creating and updating, in the on-premises storage, a B+tree for the snapshot comprising internal nodes and leaf nodes, each leaf node representing a data block of the snapshot and including a pointer to a cloud physical block address (CPBA) in the cloud/object storage where the data block has been or will be uploaded.

10. The non-transitory computer readable storage medium of claim 8 wherein uploading the metadata comprises:
packaging the metadata into one or more fixed-size metadata chunks and uploading the one or more fixed-size metadata chunks to the cloud/object storage.

11. The non-transitory computer readable storage medium of claim 8 wherein a predefined range of local physical block addresses (LPBAs) is reserved for snapshot metadata for the dataset in the on-premises storage, and wherein an identical predefined range of CPBAs is reserved for snapshot metadata for the dataset in the cloud/object storage.

12. The non-transitory computer readable storage medium of claim 11 wherein staging the metadata for the snapshot in the on-premises storage comprises:
allocating blocks for the metadata sequentially from the predefined range of LPBAs.

13. The non-transitory computer readable storage medium of claim 12 wherein uploading the metadata staged in the on-premises storage to the cloud/object storage comprises:
packaging the metadata into one or more fixed-size metadata chunks having sequential chunk identifiers from the predefined range of CPBAs and uploading the one or more fixed-size metadata chunks with the sequential chunk identifiers to the cloud/object storage, wherein the CPBA of each block of the metadata in the cloud/object storage maps directly to the LPBA of said each block of the metadata in the on-premises storage.

14. The non-transitory computer readable storage medium of claim 8 wherein staging the metadata for the snapshot in the on-premises storage comprises:
allocating blocks for the metadata randomly from a local physical block address space on the on-premises storage.

15. A computer system residing at an on-premises site including on-premises storage, the computer system comprising:
a processor; and
a non-transitory computer readable medium having stored thereon program code that, when executed, causes the processor to:
receive a snapshot of a dataset to be archived, the snapshot including data of the dataset;
package the data included in the snapshot into one or more fixed-size data chunks and uploading the one or more fixed-size data chunks to cloud/object storage;
concurrently with the packaging and the uploading, stage metadata for the snapshot in the on-premises storage; and
upon uploading all of the one or more fixed-size data chunks, upload the metadata staged in the on-premises storage to the cloud/object storage.

16. The computer system of claim 15 wherein staging the metadata for the snapshot in the on-premises storage comprises:
creating and updating, in the on-premises storage, a B+ tree for the snapshot comprising internal nodes and leaf nodes, each leaf node representing a data block of the snapshot and including a pointer to a cloud physical block address (CPBA) in the cloud/object storage where the data block has been or will be uploaded.

17. The computer system of claim 15 wherein uploading the metadata comprises:
packaging the metadata into one or more fixed-size metadata chunks and uploading the one or more fixed-size metadata chunks to the cloud/object storage.

18. The computer system of claim 15 wherein a predefined range of local physical block addresses (LPBAs) is reserved for snapshot metadata for the dataset in the on-premises storage, and wherein an identical predefined range of CPBAs is reserved for snapshot metadata for the dataset in the cloud/object storage.

Assignees

• Vmware Inc

Inventors

• Kashi Visvanathan Satish Kumar
• Sarda Pooja
• Kandambakkam Arun
• Zucca Michael
• Langouev Ilya

Classifications

• G06F11/1451 Primary

  by selection of backup contents · CPC title

• G06F16/128

  Details of file system snapshots on the file-level, e.g. snapshot creation, administration, deletion (error detection or correction of the data by redundancy in operations or in hardware G06F11/14, G06F11/16) · CPC title

• G06F16/113 Primary

  Details of archiving (lifecycle management in storage systems G06F3/0649; point-in-time backing up or restoration of persistent data G06F11/1446) · CPC title

• G06F11/1464

  for networked environments · CPC title

• G06F11/1458

  Management of the backup or restore process · CPC title