Methods to identify, handle and recover from suspect SSDS in a clustered flash array

Patent metadata

Field	Value
Publication number	US-9710317-B2
Application number	US-201514673258-A
Country	US
Kind code	B2
Filing date	Mar 30, 2015
Priority date	Mar 30, 2015
Publication date	Jul 18, 2017
Grant date	Jul 18, 2017

Abstract

A technique predicts failure of one or more storage devices of a storage array serviced by a storage system and for establishes one or more threshold conditions for replacing the storage devices. The predictive technique periodically monitors soft and hard failures of the storage devices (e.g., from Self-Monitoring, Analysis and Reporting Technology), as well as various usage counters pertaining to input/output (I/O) workloads and response times of the storage devices. A heuristic procedure may be performed that combines the monitored results to calculate the predicted failure and recommend replacement of the storage devices, using one or more thresholds based on current usage and failure patterns of the storage devices. In addition, one or more policies may be provided for replacing the storage devices in a cost-effective manner that ensures non-disruptive operation and/or replacement of the SSDs, while obviating a potential catastrophic scenario based on the usage and failure patterns of the storage devices.

First claim

What is claimed is:

1. A method comprising:
receiving a write request having data directed towards a storage container stored on a plurality of solid state storage devices (SSDs) included in a storage array connected to a node;
issuing an input/output (I/O) storage operation to store the data to a first SSD of the storage array;
in response to an I/O error detected from the I/O storage operation, incrementing a first counter in a memory included in the node, wherein the first counter is associated with a first I/O error type;
associating a set of counters with each SSD of the storage array, wherein the first counter is included in the set of counters associated with the first SSD, wherein the first I/O error type is a recovered error; and
in response to the first counter exceeding a first predetermined threshold after a first periodic interval, issuing an alert to migrate the data from the first SSD to a second SSD of the storage array, and placing the first SSD into read-only service, and wherein the first predetermined threshold is chosen such that an expected failure of the first SSD occurs after a migration period for the data.

2. The method of claim 1 wherein a second counter include in the set of counters associated with the first SSD has a second I/O error type different from the first I/O error type.

3. The method of claim 2 further comprising:
determining whether the second counter exceeds a second threshold during a second periodic interval, wherein the second I/O error type is a medium error; and
in response to determining that the second counter does not exceed the second threshold during the second periodic interval, determining whether the second counter exceeds a third threshold during a third periodic interval having a longer duration than the second periodic interval, wherein the third threshold is larger than the second threshold.

4. The method of claim 1 wherein the first counter is reset in response to not detecting an I/O error of the first I/O error type during a second periodic interval occurring after the first periodic interval, the second periodic interval having a same duration as the first periodic interval.

5. The method of claim 1 further comprising:
scheduling a staged replacement of the first SSD such that a minimum level of redundancy of the storage array is maintained during the migration period.

6. The method of claim 1 wherein the I/O storage operation is a write operation and wherein a storage medium of each SSD wears out after approximately a same number of write operations.

7. The method of claim 1 wherein the first SSD is powered down by the node in response to determining that an attribute of the first SSD indicates a power-on hours exceeds a power-on threshold.

8. The method of claim 1 wherein a second counter included in the set of counters associated with the first SSD is associated with a timeout error type of I/O error, and wherein the first SSD has multi-layer-cell flash components.

9. A method comprising:
receiving a write request having data directed towards a storage container stored on a plurality of solid state storage drives (SSDs) included in a storage array connected to a node;
issuing an input/output (I/O) storage operation to store the data to a first SSD of the storage array;
reading an attribute of the first SSD, wherein the attribute is selected from a group consisting of a number of defective blocks, a number of reserved blocks used, and a number of reassigned blocks; and
in response to determining that the attribute exceeds a threshold, wherein the threshold is normalized based on a flash component type included in the first SSD and a storage capacity of the first SSD, issuing an alert to migrate the data from the first SSD to a second SSD of the storage array, and placing the first SSD into read-only service, and wherein the predetermined threshold is chosen such that an expected failure of the first SSD occurs after a migration period for the data.

10. A system comprising:
a node of a cluster, the node having a memory connected to a processor via a bus;
a storage array connected to the node having one or more solid state drives (SSDs);
a storage input/output (I/O) stack executing on the processor of the node, the storage I/O stack configured to:
receive a write request having data directed towards a storage container stored on the storage array;
issue an I/O storage operation to a first SSD of the storage array;
in response to an I/O error detected from the I/O storage operation, increment a first counter in the memory, wherein the first counter is associated with a first type of I/O error, and wherein the first counter is included in a set of counters associated with the first SSD, wherein the first type of I/O error is a recovered error; and
in response to the first counter exceeding a first predetermined threshold after a first periodic interval, issue an alert to migrate the data from the first SSD to a second SSD of the storage array, and place the first SSD into read-only service, and wherein the first predetermined threshold is chosen such that an expected failure of the first SSD occurs after a migration period for the data.

11. The system of claim 10 wherein a second counter included in the set of counters associated with the first SSD has a second type of I/O error different from the first type of I/O error.

12. The system of claim 11 wherein the storage I/O stack is further configured to:
determine whether the second counter exceeds a second threshold during a second periodic interval, wherein the second type of I/O error is a medium error; and
in response to determining that the second counter does not exceed the second threshold during the second periodic interval, determine whether the second counter exceeds a third threshold during a third periodic interval having a longer duration than the second periodic interval, wherein the third threshold is larger than the second threshold.

13. The system of claim 10 wherein the first counter is reset in response to not detecting an I/O error of the first type of I/O error during a second periodic interval occurring after the first periodic interval, the second periodic interval having a same duration as the first periodic interval.

14. The system of claim 10 wherein the storage I/O stack is further configured to:
schedule a staged replacement of the first SSD such that a minimum level of redundancy of the storage array is maintained during the migration period.

15. The system of claim 10 wherein the first SSD is powered down by the storage I/O stack in response to determining that an attribute of the SSD indicates a power-on hours exceeds a power-on threshold.

16. The system of claim 10 wherein the I/O storage operation is a write operation and wherein the storage I/O stack is further configured to:
migrate all data from a first shelf of SSDs having the first SSD to a second shelf of SSDs having the second SSD, wherein one or more flash components of each SSD of the first shelf wears out after approximately a same number of write operations.

17. The system of claim 16 wherein the migration of the data from the first shelf to the second shelf occurs during the migration period.

18. The system of claim 10 wherein the I/O storage operation is a write operation and wherein the storage I/O stack is further configured to:
schedule a staged replacement of all SSDs included in a shelf of the storage array, wherein one or more flash components of each SSD of the shelf wears out after approximately a same num

Assignees

• Netapp Inc

Inventors

• Gupta Anish
• Mohammed Samiullah

Classifications

• G06F3/0616

  in relation to life time, e.g. increasing Mean Time Between Failures [MTBF] · CPC title

• G06F3/0688

  Non-volatile semiconductor memory arrays · CPC title

• G06F3/0647

  Migration mechanisms · CPC title

• G06F11/0772

  Means for error signaling, e.g. using interrupts, exception flags, dedicated error registers · CPC title

• G06F11/0793

  Remedial or corrective actions (recovery from an exception in an instruction pipeline G06F9/3861; by retry G06F11/1402; for recovering from a failure of a protocol instance or entity H04L69/40) · CPC title