Capacity exhaustion prevention for distributed storage

Patent metadata

Field	Value
Publication number	US-10067696-B2
Application number	US-201615186576-A
Country	US
Kind code	B2
Filing date	Jun 20, 2016
Priority date	Dec 18, 2015
Publication date	Sep 4, 2018
Grant date	Sep 4, 2018

Abstract

Systems and processes to prevent storage capacity exhaustion within distributed storage systems using chunk-based storage. The systems and processes control capacity by classifying various types of storage chunks into groups and impose used capacity thresholds on chunk groups.

First claim

What is claimed is:

1. A method for use with a storage node of a distributed storage system having a plurality of storage nodes, the storage node having a plurality of storage devices, the method comprising:
maintaining statistics on storage capacity used within the storage devices;
receiving a request to allocate a storage chunk having a given chunk type;
using the chunk type to determine a chunk group from a plurality of chunk groups, each of the chunk groups having an assigned used capacity threshold, wherein the plurality of chunk groups include a user group associated with user data and a system group associated with system data;
determining a node used capacity based on the statistics; and
allocating a new storage chunk within the plurality of storage devices only if the node used capacity is less than the chunk group used capacity threshold, wherein a used capacity threshold assigned to the system group is greater than a used capacity threshold assigned to the user group such that the system can store new system data but not new user data when the node used capacity exceeds the used capacity threshold assigned to the user group.

2. The method of claim 1 further comprising:
returning an error indicating insufficient storage capacity if the node used capacity is greater than the chunk group used capacity threshold.

3. The method of claim 1 wherein the used capacity threshold assigned to the system group is 100%.

4. The method of claim 1 wherein the system data includes erasure coding data and XOR data.

5. The method of claim 1 further including using hysteresis to prevent frequent switching between accepting new user data and not accepting new user data.

6. The method of claim 5 wherein the chunk groups further include a replication group associated with replication data.

7. The method of claim 6 wherein a used capacity threshold assigned to the replication group is greater than a used capacity threshold assigned to the user group such that the system can replicate data but not store new user data when the node used capacity exceeds the used capacity threshold assigned to the user group.

8. A distributed storage system, comprising:
a plurality of storage nodes each having one or more storage devices and being configured to:
maintain statistics on storage capacity used within the storage devices;
receive a request to allocate a storage chunk having a given chunk type;
use the chunk type to determine a chunk group from a plurality of chunk groups, each of the chunk groups having an assigned used capacity threshold, wherein the chunk groups include a user group associated with user data and a system group associated with system data;
determine a node used capacity based on the statistics; and
allocate a new storage chunk within the plurality of storage devices only if the node used capacity is less than the chunk group used capacity threshold, wherein a used capacity threshold assigned to the system group is greater than a used capacity threshold assigned to the user group such that the system can store new system data but not new user data when the node used capacity exceeds the used capacity threshold assigned to the user group.

9. The distributed storage system of claim 8 wherein each of the plurality of storage nodes is further configured to return an error indicating insufficient storage capacity if the node used capacity is greater than the chunk group used capacity threshold.

10. The distributed storage system of claim 8 wherein the used capacity threshold assigned to the system group is 100%.

11. The distributed storage system of claim 8 wherein each of the plurality of storage nodes is further configured to use hysteresis to prevent frequent switching between accepting new user data and not accepting new user data.

12. The distributed storage system of claim 11 wherein the chunk groups further include a replication group associated with replication data.

13. The distributed storage system of claim 12 wherein a used capacity threshold assigned to the replication group is greater than a used capacity threshold assigned to the user group such that the system can replicate data but not store new user data when the node used capacity exceeds the used capacity threshold assigned to the user group.

Assignees

• Emc Corp

Inventors

• Malygin Mikhail
• Danilov Mikhail
• Wang Chen
• Tchoub Ivan
• Fomin Andrey

Classifications

• G06F3/0644

  Management of space entities, e.g. partitions, extents, pools · CPC title

• G06F3/0605

  by facilitating the interaction with a user or administrator · CPC title

• G06F3/0649

  Lifecycle management · CPC title

• G06F3/067 Primary

  Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS] · CPC title

• G06F3/0619 Primary

  in relation to data integrity, e.g. data losses, bit errors · CPC title