Selective deduplication

Patent metadata

Field	Value
Publication number	US-2020125536-A1
Application number	US-201916716759-A
Country	US
Kind code	A1
Filing date	Dec 17, 2019
Priority date	Oct 18, 2012
Publication date	Apr 23, 2020
Grant date	—

Abstract

Methods and apparatuses for performing selective deduplication in a storage system are introduced here. Techniques are provided for determining a probability of deduplication for a data object based on a characteristic of the data object and performing a deduplication operation on the data object in the storage system prior to the data object being stored in persistent storage of the storage system if the probability of deduplication for the data object has a specified relationship to a specified threshold.

First claim

1 . A method, comprising:
evaluating characteristics of a data object to determine a deduplication priority of the data object based upon a projected likelihood that deduplication of the data object will reduce storage consumption within a storage device; and performing post-processing deduplication for the data object based upon the deduplication priority being less than a deduplication probability threshold.

2 . The method of claim 1 , comprising:
performing the post-processing deduplication subsequent to the data object being stored within the storage device.

3 . The method of claim 1 , comprising:
determining the deduplication priority based upon whether aggregated statistical information indicates that deduplication of data objects, having the characteristics of the data object, previously provided deduplication benefits.

4 . The method of claim 1 , comprising:
defining the deduplication probability threshold based upon predetermined performance metrics.

5 . The method of claim 1 , comprising:
performing in-line deduplication for data objects before the data objects are stored within the storage device based upon the data objects having deduplication priorities greater than the deduplication probability threshold.

6 . The method of claim 1 , comprising:
performing the post-processing deduplication based upon a determination that a current system load demand is below a threshold.

7 . The method of claim 1 , comprising:
performing the post-processing deduplication as a background operation.

8 . A computing device comprising:
a memory having stored thereon instructions for performing a method; and a processor coupled to the memory, the processor configured to execute the instructions to cause the processor to:
evaluate characteristics of a data object to determine a deduplication priority of the data object based upon a projected likelihood that deduplication of the data object will reduce storage consumption within a storage device; and
determining a deduplication probability threshold based upon performance metrics; and
perform post-processing deduplication for the data object based upon the deduplication priority being less than the deduplication probability.

9 . The computing device of claim 8 , wherein the instructions cause the processor to:
determine the deduplication priority based upon a size characteristic of the data object.

10 . The computing device of claim 8 , wherein the instructions cause the processor to:
determine the deduplication priority based upon a data object type of the data object.

11 . The computing device of claim 8 , wherein the instructions cause the processor to:
determine the deduplication priority based upon a last modified timestamp of the data object.

12 . The computing device of claim 8 , wherein the instructions cause the processor to:
determine the deduplication priority based upon an update frequency of the data object.

13 . The computing device of claim 8 , wherein the instructions cause the processor to:
determine the deduplication priority based upon statistic information derived from a previous deduplication operation.

14 . The computing device of claim 8 , wherein the instructions cause the processor to:
determine the deduplication priority based upon at least one a read throughput, a write throughput, a read response time, a write response time, network utilization, and a performance characteristic.

15 . A non-transitory machine-readable storage media having stored thereon instructions, for performing a method, which causes a processor to:
evaluate characteristics of a data object against aggregated statistical information of whether deduplication of data objects with the characteristics resulted in storage consumption savings to determine a deduplication priority of the data object based upon a projected likelihood that deduplication of the data object will reduce storage consumption within a storage device; and perform post-processing deduplication for the data object based upon the deduplication priority being less than a deduplication probability threshold.

16 . The non-transitory machine-readable storage media of claim 15 , wherein the instructions cause the processor to:
determine the deduplication priority based upon a data object type of the data object.

17 . The non-transitory machine-readable storage media of claim 15 , wherein the instructions cause the processor to:
determine the deduplication priority based upon a last modified timestamp of the data object.

18 . The non-transitory machine-readable storage media of claim 15 , wherein the instructions cause the processor to:
determine the deduplication priority based upon an update frequency of the data object.

19 . The non-transitory machine-readable storage media of claim 15 , wherein the instructions cause the processor to:
determine the deduplication priority based upon statistic information derived from a previous deduplication operation.

20 . The non-transitory machine-readable storage media of claim 15 , wherein the instructions cause the processor to:
determine the deduplication priority based upon a size characteristic of the data object.

Assignees

• Netapp Inc

Inventors

• Mallaiah Damarugendra
• Basak Jayanta

Classifications

• G06F3/0641

  De-duplication techniques · CPC title

• G06F16/1748 Primary

  De-duplication implemented within the file system, e.g. based on file segments (de-duplication techniques in storage systems for the management of data blocks G06F3/0641) · CPC title

• G06F16/2358

  Change logging, detection, and notification (replication G06F16/27) · CPC title

• G06F3/067

  Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS] · CPC title

• G06F3/0608 Primary

  Saving storage space on storage systems · CPC title