Snapshot metadata arrangement for efficient cloud integrated data management

Patent metadata

Field	Value
Publication number	US-11768803-B2
Application number	US-202017029920-A
Country	US
Kind code	B2
Filing date	Sep 23, 2020
Priority date	Oct 28, 2016
Publication date	Sep 26, 2023
Grant date	Sep 26, 2023

Abstract

A storage appliance arranges snapshot data and snapshot metadata into different structures, and arranges the snapshot metadata to facilitate efficient snapshot manipulation, which may be for snapshot management or snapshot restore. The storage appliance receives snapshots according to a forever incremental configuration and arranges snapshot metadata into different types of records. The storage appliance stores these records in key-value stores maintained for each defined data collection (e.g., volume). The storage appliance arranges the snapshot metadata into records for inode information, records for directory information, and records that map source descriptors of data blocks to snapshot file descriptors. The storage appliance uses a locally generated snapshot identifier as a key prefix for the records to conform to a sort constrain of the key-value store, which allows the efficiency of the key-value store to be leveraged. The snapshot metadata arrangement facilitates efficient snapshot restore, file restore, and snapshot reclamation.

First claim

What is claimed is:

1. A method for managing snapshots for backup and restore operations, the method implemented by a computing device and comprising:
identifying a key-value store for a data collection in one or more file systems maintained by one or more data source devices based on a data collection identifier included in a snapshot restore request, the snapshot restore request further comprises a source snapshot identifier identifying a snapshot;
retrieving a first plurality of records from the key-value store using a local snapshot identifier as a key prefix for each of the first plurality of records, wherein the first plurality of records comprises at least data map records and the local snapshot identifier is mapped to the source snapshot identifier in metadata for the data collection;
assembling data blocks from a snapshot data file into a data buffer according to offsets in the data map records that map to locations in the snapshot data file, wherein the snapshot data file is identified using the local snapshot identifier, comprises an aggregation of file data for files associated with the snapshot, and is retrieved from one or more cloud storage devices; and
sending via a data storage network at least the data buffer and information from the data map records to a backup application hosted by a restore target identified in the snapshot restore request and configured to reassemble the data blocks into the files to at least partially satisfy the snapshot restore request.

2. The method of claim 1 , wherein the first plurality of records further comprises inode records and directory records and the method further comprises sending the inode records and the directory records to the restore target when the snapshot is a baseline snapshot.

3. The method of claim 1 , further comprising determining when the snapshot is a baseline snapshot based on a comparison of the local snapshot identifier with another local snapshot identifier corresponding to the baseline snapshot in a data structure associated with the data collection and storing parent-child relationships for one or more snapshots for the data collection.

4. The method of claim 3 , further comprising retrieving a second plurality of records for each of the snapshots using another local identifier for each of the snapshots identified in the data structure as another key prefix in the key-value store, when the snapshot is an incremental snapshot.

5. The method of claim 4 , further comprising sending the second plurality of records to the restore target after the second plurality of records is merged with the first plurality of records.

6. The method of claim 5 , further comprising merging valid data ranges into a restore data map prior to assembling the data blocks into the data buffer based on the restore data map and sending the data buffer to the restore target.

7. A non-transitory machine-readable medium having stored thereon instructions for managing snapshots for backup and restore operations comprising machine executable code that, when executed by at least one machine, causes the machine to:
identify a key-value store for a data collection based on a data collection in one or more file systems maintained by one or more data source devices based on a data collection identifier included in a snapshot restore request, wherein the snapshot restore request further comprises a source snapshot identifier identifying a snapshot;
retrieve a first plurality of records from the key-value store using a local snapshot identifier as a key prefix for each of the first plurality of records, wherein the first plurality of records comprises at least data map records and the local snapshot identifier is mapped to the source snapshot identifier in metadata for the data collection;
assemble data blocks from a snapshot data file into a data buffer according to offsets in the data map records that map to locations in the snapshot data file; and
send via a data storage network the data buffer to a backup application hosted by a restore target identified in the snapshot restore request and configured to reassemble the data blocks into files associated with the snapshot to satisfy the snapshot restore request, wherein the snapshot data file is identified using the local snapshot identifier, comprises an aggregation of file data for the files, and is retrieved from one or more cloud storage devices.

8. The non-transitory machine readable medium of claim 7 , wherein the first plurality of records further comprises inode records and directory records and the machine executable code when executed by the machine further causes the machine to send the inode records and the directory records to the restore target when the snapshot is a baseline snapshot.

9. The non-transitory machine readable medium of claim 7 , wherein the machine executable code when executed by the machine further causes the machine to determine when the snapshot is a baseline snapshot based on a comparison of the local snapshot identifier with another local snapshot identifier corresponding to the baseline snapshot in a data structure associated with the data collection and storing parent-child relationships for one or more snapshots for the data collection.

10. The non-transitory machine readable medium of claim 9 , wherein the machine executable code when executed by the machine further causes the machine to retrieve a second plurality of records for each of the snapshots using another local identifier for each of the snapshots identified in the data structure as another key prefix in the key-value store, when the snapshot is an incremental snapshot.

11. The non-transitory machine readable medium of claim 10 , wherein the machine executable code when executed by the machine further causes the machine to send the second plurality of records to the restore target after the second plurality of records is merged with the first plurality of records.

12. The non-transitory machine readable medium of claim 7 , wherein the machine executable code when executed by the machine further causes the machine to merge valid data ranges into a restore data map prior to assembling the data blocks into the data buffer based on the restore data map and sending the data buffer to the restore target.

13. A computing device, comprising:
a memory containing a non-transitory machine-readable medium comprising machine executable code having stored thereon instructions for managing snapshots to facilitate efficient backup and restore operations in data storage networks; and
a processor coupled to the memory, the processor configured to execute the machine executable code to cause the processor to:
retrieve a first plurality of records from a key-value store for a data collection using a local snapshot identifier as a key prefix in response to a snapshot restore request comprising a source snapshot identifier identifying the data collection, wherein data collection is hosted in one or more file systems maintained by one or more data source devices and the first plurality of records comprises at least data map records and the local snapshot identifier is mapped to the source snapshot identifier;
assemble data blocks from a snapshot data file into a data buffer according to offsets in the data map records that map to locations in a snapshot data file, wherein the snapshot data file is identified using the local snapshot identifier, comprises an aggregation of file data for files associated with the snapshot, and is retrieved from one or more cloud storage devices; and
send via a data storage network at least the data buffer and the data map records to a backup application hosted by a restor

Assignees

• Netapp Inc

Inventors

• Jain Sharad
• Tapaswi Hrishikesh Ashok
• Zheng Ling
• Bhat Onkar
• Kaushik Akhil

Classifications

• G06F16/128 Primary

  Details of file system snapshots on the file-level, e.g. snapshot creation, administration, deletion (error detection or correction of the data by redundancy in operations or in hardware G06F11/14, G06F11/16) · CPC title

• G06F11/1448 Primary

  Management of the data involved in backup or backup restore · CPC title

• G06F11/1464

  for networked environments · CPC title

• G06F11/1469

  Backup restoration techniques · CPC title

• G06F16/13

  File access structures, e.g. distributed indices (arrangements of input from, or output to, record carriers G06F3/06) · CPC title