Recovery of clustered storage systems

Patent metadata

Field	Value
Publication number	US-12282676-B2
Application number	US-202318116740-A
Country	US
Kind code	B2
Filing date	Mar 2, 2023
Priority date	Feb 28, 2022
Publication date	Apr 22, 2025
Grant date	Apr 22, 2025

Abstract

A cluster storage system takes snapshots that are consistent across all storage nodes. The storage system can nearly instantaneously promote a set of consistent snapshots to their respective base volumes to restore the base volumes to be the same as the snapshots. Given these two capabilities, users can restore the system to a recovery point of the user's choice, by turning off storage service I/O, promoting the snapshots constituting the recovery point, rebooting their servers, and resuming storage service I/O.

First claim

What is claimed is:

1. A recovery process for a storage system comprising:
creating, for one or more recovery points, a plurality of snapshot sets respectively in a plurality of storage nodes of the storage system, each snapshot set of the snapshot sets corresponding to a recovery point of the one or more recovery points and containing snapshots of every volume that a storage node of the plurality of storage nodes owns or maintains;
assigning, for the one or more recovery points, a generation number to each snapshot in a corresponding snapshot set associated with a respective recovery point;
receiving, from a user of the storage system, a selection of a recovery point from the one or more recovery points;
suspending one or more storage services of the storage system;
in response to receiving the selection of the recovery point, in each storage node of the plurality of storage nodes, promoting the snapshots in the snapshot set corresponding to the recovery point selected by altering metadata associated with each storage node to point to respective data with a generation number in a range between a generation number at creation of the volumes and the generation number assigned to the selected recovery point;
rebooting the storage nodes; and
resuming the one or more storage services of the storage system.

2. The process of claim 1 , wherein creating the snapshot sets comprises:
receiving, from the user of the storage system, a selection of a schedule for creation of the snapshot sets;
synchronizing the storage nodes to identify a first time in the schedule; and
creating, at the first time, the snapshot sets that correspond to a first of the recovery points, the first recovery point corresponding to a state of the storage system at the first time.

3. The process of claim 2 , wherein creating the snapshot set further comprises:
causing the storage nodes to identify a second time in the schedule; and
at the second time, causing the storage nodes to create the snapshot sets that correspond to a second of the recovery points, the second recovery point corresponding to a state of the storage system at the second time.

4. The process of claim 2 , wherein synchronizing the storage nodes comprises executing a Network Time Protocol (NTP) process on the storage nodes through a network interconnecting the storage nodes.

5. The process of claim 1 , further comprising, after the suspending of the storage services, creating a plurality of new snapshot sets respectively in the storage nodes of the storage system, the new snapshot sets corresponding to a new recovery point and capturing a state of the storage system when the recovery process was performed.

6. The process of claim 1 , wherein the creating of the snapshot sets respectively in the storage nodes of the storage system, consists of each of the storage nodes, without copying or moving data of the volumes, creating a metadata structure that prevents the storage node from deleting from the physical storage any of the data required for the snapshots.

7. The process of claim 6 , wherein the promoting of the snapshots consists of each of the storage nodes, without copying or moving the data of the volumes in physical storage, modifying a metadata structure.

8. A cluster storage system comprising:
a plurality of storage nodes; and
a network interconnecting the storage nodes, wherein the storage nodes are configured to perform a process including:
creating, for one or more recovery points, a plurality of snapshot sets respectively in the storage nodes, each snapshot set of the snapshot sets corresponding to a recovery point of the one or more recovery points and containing snapshots of every volume that a storage node of the plurality of storage nodes owns or maintains;
assigning, for the one or more recovery points, a generation number to each snapshot in a corresponding snapshot set associated with a respective recovery point;
receiving a request to roll back the storage system to one of the one or more recovery points selected by a user of the cluster storage system from the one or more recovery points;
suspending storage services of the storage system;
in response to receiving the selection of the recovery point, in each storage node of the plurality of storage nodes, promoting the snapshots in the snapshot set corresponding to the recovery point selected by altering metadata associated with each storage node to point to respective data with a generation number in a range between a generation number at creation of the volumes and the generation number assigned to the selected recovery point;
rebooting the storage nodes; and
resuming the storage service of the storage system.

9. The cluster storage system of claim 8 , wherein each of the storage nodes comprises:
a server;
a backend storage device; and
a storage processing unit resident in the server and connected to control the backend storage to provide the storage services that target any of the volumes that the storage node owns or maintains.

10. The cluster storage system of claim 9 , wherein in each storage node, the storage processing unit comprises a card plugged into a bus of the server.

11. The cluster storage system of claim 8 , wherein the cluster storage system is configured to synchronize the storage nodes by executing a Network Time Protocol (NTP) process through the network interconnecting the storage nodes.

12. A system comprising:
at least one processor; and
a memory coupled to the at least one processor, the memory storing instructions that, when executed by the at least one processor, cause the system to perform steps comprising:
creating, for one or more recovery points, a plurality of snapshot sets respectively in a plurality of storage nodes of the storage system, each snapshot set of the snapshot sets corresponding to a recovery point of the one or more recovery points and containing snapshots of every volume that a storage node of the plurality of storage nodes owns or maintains;
assigning, for the one or more recovery points, a generation number to each snapshot in a corresponding snapshot set associated with a respective recovery point;
receiving, from a user of the storage system, a selection of a recovery point from the one or more recovery points selecting one of the one or more recovery points;
suspending one or more storage services of the storage system;
in response to receiving the selection of the recovery point, in each storage node of the plurality of storage nodes, promoting the snapshots in the snapshot set corresponding to the recovery point selected by altering metadata associated with each storage node to point to respective data with a generation number in a range between a generation number at creation of the volumes and the generation number assigned to the selected recovery point;
rebooting the storage nodes; and
resuming the one or more storage services of the storage system.

13. The system of claim 12 , wherein creating the snapshot sets comprises:
receiving, from the user of the storage system, a selection of a schedule for creation of the snapshot sets;
synchronizing the storage nodes to identify a first time in the schedule; and
creating, at the first time, the snapshot sets that correspond to a first of the recovery points, the first recovery point corresponding to a state of the storage system at the first time.

14. The system of claim 13 , wherein creating the snapshot set further comprises:
causing the storage nodes to identify a second time in the schedule; and
at the second time, causing the storage nodes to create the snapshot sets that correspo

Assignees

• Nvidia Corp

Inventors

• Nazari Siamak
• Dejong David
• Murthy Srinivasa
• Askarian Namaghi Shayan
• Tamma Roopesh

Classifications

• G06F3/0623

  in relation to content · CPC title

• G06F3/0683

  Plurality of storage devices · CPC title

• G06F3/067

  Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS] · CPC title

• G06F11/1469

  Backup restoration techniques · CPC title

• G06F3/065 Primary

  Replication mechanisms · CPC title