Performing reconciliation on a segmented de-duplication index

Patent metadata

Field	Value
Publication number	US-10845994-B1
Application number	US-201715664185-A
Country	US
Kind code	B1
Filing date	Jul 31, 2017
Priority date	Jul 31, 2017
Publication date	Nov 24, 2020
Grant date	Nov 24, 2020

Abstract

A technique accesses a non-resident segment and a resident segment of a segmented de-duplication index, the resident segment being currently loaded into primary memory from secondary storage for data block de-duplication, and the non-resident segment not being currently loaded into the primary memory from the secondary storage for de-duplication. The technique further discovers that a digest of a non-resident digest entry of the non-resident segment and a digest of a resident digest entry of the resident segment are duplicates. The non-resident digest entry includes a first reference to a first location of the secondary storage that holds a first data block copy, and the resident digest entry includes a second reference to a second location of the secondary storage that holds a second data block copy. The technique further performs reconciliation that conforms the non-resident segment and the resident segment of the index to reference only data block copy.

First claim

What is claimed is:

1. In data storage equipment, a method of managing a de-duplication index having segments containing digest entries that reference data blocks residing within secondary storage, the method comprising:
accessing a non-resident segment of the de-duplication index and a resident segment of the de-duplication index, the resident segment being currently loaded into primary memory from the secondary storage for data block de-duplication, and the non-resident segment not being currently loaded into the primary memory from the secondary storage for data block de-duplication;
in response to accessing the non-resident segment and the resident segment, discovering that (i) a digest of a non-resident digest entry of the non-resident segment and (ii) a digest of a resident digest entry of the resident segment are duplicates, the non-resident digest entry including a first reference to a first storage location of the secondary storage that holds a first copy of a particular data block, and the resident digest entry including a second reference to a second storage location of the secondary storage that holds a second copy of the particular data block; and
in response to discovering that (i) the digest of the non-resident digest entry of the non-resident segment and (ii) the digest of the resident digest entry of the resident segment are duplicates, performing a reconciliation operation that conforms the non-resident segment and the resident segment of the de-duplication index to reference only one of the first copy and the second copy of the particular data block;
wherein performing the reconciliation operation includes:
from the de-duplication index, eliminating one of the first reference to the first storage location of the secondary storage and the second reference to the second storage location of the secondary storage;

wherein the method further comprises:
prior to performing the reconciliation operation, performing a resident segment evaluation operation to determine whether the resident segment is open to receiving new digest entries or not open to receiving new digest entries;

wherein eliminating includes:
deleting the resident digest entry from the resident segment when a result of the resident segment evaluation operation indicates that the resident segment is open to receiving new digest entries, and
replacing, within the non-resident digest entry, the first reference to the first storage location with the second reference to the second storage location when the result of the resident segment evaluation operation indicates that the resident segment is not open to receiving new digest entries.

2. A method as in claim 1 wherein discovering includes:
performing a comparison operation between the digest of a non-resident digest entry of the non-resident segment and the digest of a resident digest entry of the resident segment to determine whether the digests are duplicates.

3. A method as in claim 2 , further comprising:
prior to performing the comparison operation between the digest of the non-resident digest entry of the non-resident segment and the digest of the resident digest entry of the resident segment, filtering the digest of the non-resident digest entry against a predictive filter to predict whether the resident segment indexes a copy of the particular data block.

4. A method as in claim 1 wherein the data storage equipment is operative to perform data storage operations to write data to and read data from the secondary storage via datasets; and
wherein the method further comprises:
performing a de-duplication operation that shares one of the first copy and the second copy of the particular data block among different datasets.

5. A method as in claim 4 wherein initially a first dataset includes the first reference to the first storage location of the secondary storage that holds the first copy of the particular data block;
wherein initially a second dataset includes the second reference to the second storage location of the secondary storage that holds the second copy of the particular data block;
wherein performing the de-duplication operation includes:
accessing a back reference of the first copy of the particular data block to identify the first dataset among the dataset, and
upon identifying the first dataset, replacing the first reference with the second reference within the first dataset to share the second copy of the particular data block among both the first dataset and the second dataset.

6. A method as in claim 5 wherein performing the de-duplication operation further includes:
in response to replacing the first reference with the second reference within the first dataset, deleting the first copy of the particular data block from the secondary storage to reclaim memory space within the secondary storage.

7. A method as in claim 5 , further comprising:
in response to replacing the first reference with the second reference within the first dataset, leaving the first copy of the particular data block in place until the first copy of the particular data block is rewritten with new dataset content.

8. A method as in claim 5 , further comprising:
in response to replacing the first reference with the second reference within the first dataset, leaving the first copy of the particular data block in place until the first copy of the particular data block is deleted in response to dataset deletion.

9. In data storage equipment, a method of managing a de duplication index having segments containing digest entries that reference data blocks residing within secondary storage, the method comprising:
accessing a non-resident segment of the de-duplication index and a resident segment of the de-duplication index, the resident segment being currently loaded into primary memory from the secondary storage for data block de-duplication, and the non-resident segment not being currently loaded into the primary memory from the secondary storage for data block de-duplication;
in response to accessing the non-resident segment and the resident segment, discovering that (i) a digest of a non-resident digest entry of the non resident segment and (ii) a digest of a resident digest entry of the resident segment are duplicates, the non-resident digest entry including a first reference to a first storage location of the secondary storage that holds a first copy of a particular data block, and the resident digest entry including a second reference to a second storage location of the secondary storage that holds a second copy of the particular data block; and
in response to discovering that (i) the digest of the non-resident digest entry of the non resident segment and (ii) the digest of the resident digest entry of the resident segment are duplicates, performing a reconciliation operation that conforms the non-resident segment and the resident segment of the de-duplication index to reference only one of the first copy and the second copy of the particular data block;
wherein performing the reconciliation operation includes:
from the de-duplication index, eliminating one of the first reference to the first storage location of the secondary storage and the second reference to the second storage location of the secondary storage;

wherein the method further comprises:
prior to performing the reconciliation operation, performing a resident segment evaluation operation to determine whether the resident segment is open to receiving new digest entries or not open to receiving new digest entries;

wherein a result of the resident segment evaluation operation indicates that the resident segment is open to receiving new digest entries; and
wherein eliminating includes:
in response to the result of the resi

Assignees

• Emc Ip Holding Co Llc

Inventors

• Usvyatsky Ilya
• Dalmatov Nickolay Alexandrovich

Classifications

• G06F3/067

  Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS] · CPC title

• G06F3/0652

  Erasing, e.g. deleting, data cleaning, moving of data to a wastebasket · CPC title

• G06F3/0641

  De-duplication techniques · CPC title

• G06F3/0608 Primary

  Saving storage space on storage systems · CPC title

• G06F16/1748 Primary

  De-duplication implemented within the file system, e.g. based on file segments (de-duplication techniques in storage systems for the management of data blocks G06F3/0641) · CPC title