Generating copies through interoperability between a data storage management system and appliances for data storage and deduplication

Patent metadata

Field	Value
Publication number	US-11681587-B2
Application number	US-202117214628-A
Country	US
Kind code	B2
Filing date	Mar 26, 2021
Priority date	Nov 27, 2018
Publication date	Jun 20, 2023
Grant date	Jun 20, 2023

Abstract

Illustrative storage manager and media agent are enhanced to interoperate with deduplication appliances. Advantages are realized when making secondary and tertiary copies and also when restoring from a deduplication appliance. Tiered indexing minimizes how much data is retained and stored at media agents. Tiered indexing enables media agents to efficiently extract needed information from deduplication appliances to make tertiary copies and to restore backed up copies. Interoperability techniques include media agents generating separate data streams to the deduplication appliance. Each data stream carries a different kind of data, e.g., payload data, metadata content, or high-level index information. On initial backup, the media agent instructs the deduplication appliance to deduplicate the payload data stream but not the other data streams, thus intelligently applying resources to data most likely to benefit from deduplication. For tertiary copies (copies of pre-existing copies at the deduplication appliance), the media agent avoids handling payload data altogether.

First claim

What is claimed is:

1. A computer-implemented method comprising:
backing up primary data into one or more secondary copies to be stored at a deduplication appliance that comprises one or more data storage devices and wherein the deduplication appliance is capable of performing deduplication;
wherein the backing up comprises:
by a first computing device, generating without deduplication a first data stream transmitted to the deduplication appliance, wherein the first data stream comprises first data chunks and does not comprise second data chunks and third data chunks,
wherein each first data chunk comprises payload data from the primary data being backed up, and
wherein the first computing device comprises one or more processors and computer memory;
by the first computing device, generating without deduplication a second data stream transmitted to the deduplication appliance, wherein the second data stream comprises the second data chunks and does not comprise the first data chunks and the third data chunks,
wherein each second data chunk comprises metadata for the primary data being backed up and further wherein each second data chunk points to one or more first data chunks of the first data stream;
by the first computing device, generating without deduplication a third data stream transmitted to the deduplication appliance, wherein the third data stream comprises the third data chunks and does not comprise the first data chunks and the second data chunks,
wherein each third data chunk points to a corresponding second data chunk of the second data stream; and
by the first computing device, instructing the deduplication appliance: to apply deduplication to the first data chunks of the first data stream, which results in deduplicated first data chunks, to store the deduplicated first data chunks at the deduplication appliance, to store the second data chunks of the second data stream at the deduplication appliance without applying deduplication thereto, and to store the third data chunks of the third data stream at the deduplication appliance without applying deduplication thereto.

2. The method of claim 1 further comprising:
by the first computing device, storing in an associated index contents of the third data chunks which point to the corresponding second data chunks of the second data stream.

3. The method of claim 1 , further comprising:
by the first computing device, storing in an index, which is associated with the first computing device, contents of the third data chunks which point to the corresponding second data chunks of the second data stream,
wherein the first computing device adds to the index the metadata for the primary data from the second data chunks.

4. The method of claim 1 , wherein the first computing device is configured to transmit the first data stream, the second data stream, and the third data stream without deduplication to the deduplication appliance.

5. The method of claim 1 , wherein a given second data chunk includes an offset of a corresponding first data chunk within the first data stream.

6. The method of claim 1 , wherein a given third data chunk includes an offset of the corresponding second data chunk within the second data stream.

7. The method of claim 1 , wherein the first computing device executes a media agent that generates the first data stream, the second data stream, and the third data stream, and also executes a data agent that accesses the primary data being backed up; and
wherein the first computing device processes the primary data being backed up for deduplication by and storage at the deduplication appliance as the one or more secondary copies.

8. The method of claim 1 further comprising:
in response to instructions to restore a first secondary copy from the deduplication appliance:
by the first computing device, extracting from an associated index, information about the first secondary copy, wherein the information comprises pointers to a second data chunk stored without deduplication at the deduplication appliance, wherein the second data chunk comprises metadata for the first secondary copy and further comprises one or more pointers to corresponding one or more first data chunks stored with deduplication at the deduplication appliance, and wherein a given first data chunk comprises payload data of the first secondary copy;
by the first computing device, causing the deduplication appliance to transmit to the first computing device a fourth data stream that comprises the second data chunk;
by the first computing device, causing the deduplication appliance to transmit to the first computing device a fifth data stream that comprises the one or more first data chunks pointed to by the one or more pointers of the second data chunk, wherein the deduplication appliance rehydrates at least some of the first data chunks before transmitting the fifth data stream to the first computing device; and
by the first computing device processing the first data chunks and the metadata in the second data chunk to generate a sixth data stream that restores the first secondary copy into primary data.

9. The method of claim 1 further comprising:
in response to instructions to restore a first secondary copy from the deduplication appliance:
by the first computing device, extracting from an associated index a second data chunk, wherein the second data chunk comprises metadata for the first secondary copy and further comprises one or more pointers to corresponding one or more first data chunks stored with deduplication at the deduplication appliance, and wherein a given first data chunk comprises payload data of the first secondary copy;
by the first computing device, causing the deduplication appliance to transmit to the first computing device a fourth data stream that comprises the one or more first data chunks pointed to by the one or more pointers of the second data chunk, wherein at least some of the first data chunks are rehydrated by the deduplication appliance before transmitting the fourth data stream to the first computing device; and
by the first computing device processing the first data chunks and the metadata in the second data chunk to generate a sixth data stream that restores the first secondary copy into primary data.

10. The method of claim 1 , wherein a storage manager instructs the first computing device to process the primary data being backed up for further processing by and storage at the deduplication appliance as the one or more secondary copies, and wherein the storage manager executes on one of: the first computing device and a second computing device distinct from the first computing device.

11. A system comprising:
a first computing device comprising one or more hardware processors and computer memory;
wherein the first computing device is programmed to:
generate without deduplication a first data stream transmitted to a deduplication appliance, wherein the deduplication appliance comprises one or more data storage devices and is capable of performing deduplication, wherein the first data stream comprises first data chunks and does not comprise second data chunks and third data chunks,
wherein each first data chunk comprises payload data from primary data being backed up into one or more secondary copies that are to be stored at the deduplication appliance;
generate without deduplication a second data stream transmitted to the deduplication appliance, wherein the second data stream comprises the second data chunks and does not comprise the first data chunks and the third data chunks,
wherein each second data chunk comprises metadata for the primary data being backed up and further wherein each second data c

Assignees

• Commvault Systems Inc

Inventors

• Haridas Ganesh
• Vijayan Manoj Kumar

Classifications

• G06F2201/84

  Using snapshots, i.e. a logical point-in-time copy of the data · CPC title

• G06F11/1453 Primary

  using de-duplication of the data · CPC title

• G06F11/1458

  Management of the backup or restore process · CPC title

• G06F16/174

  Redundancy elimination performed by the file system (error detection or correction of the data by redundancy in operations G06F11/14) · CPC title