Handling node failure in multi-node data storage systems

Patent metadata

Field	Value
Publication number	US-10452502-B2
Application number	US-201815877405-A
Country	US
Kind code	B2
Filing date	Jan 23, 2018
Priority date	Jan 23, 2018
Publication date	Oct 22, 2019
Grant date	Oct 22, 2019

Abstract

A multi-node data storage system comprises a first data storage system having an owner node and a backup node in a first location coupled to a second data storage system having an owner node and a backup node in a second location. Each storage system includes a copy of the same data volume. A failure of a node of the multi-node storage system is detected. An outstanding write request to the first storage system is identified. If the owner node in the first storage system fails, it is determined whether the outstanding write corresponds to a host write to the backup node of the first storage system. If so, a retransmission message is sent to the second storage system. Otherwise, the data region associated with the outstanding write request is read from the first storage system, and a resynchronization message is sent to the second storage system.

First claim

What is claimed is:

1. In a multi-node storage system comprising a first data storage system having an owner node and a backup node in a first location, and a second data storage system having an owner node and a backup node in a second location, wherein the first data storage system is communicatively coupled to the second data storage system and each of the first and second data storage systems includes a copy of the same data volume, a method comprising:
detecting failure of a node of the multi-node data storage system;
identifying an outstanding write request to the first data storage system;
wherein, if the failed node is the owner node of the first data storage system, the method further comprises:
determining whether the outstanding write request corresponds to a host write to the backup node of the first data storage system,
if the outstanding write request corresponds to a host write to the backup node of the first data storage system, sending a retransmission message of the outstanding write request to a node of the second data storage system, the retransmission message comprising an indicator that the write request is a retransmission, the data to be written and the region in the data volume where the data is to be written; or
if the outstanding write request does not correspond to a host write to the backup node of the first data storage system, reading the data region associated with the outstanding write request from the first data storage system, and sending a resynchronization message to a node of the second data storage system, the resynchronization message comprising an indicator that the write request is a resynchronization, the data read from the first data storage system to be written and the region in the data volume where the data is to be written;

receiving a write request message, wherein the type of write request message comprises one of a resynchronization message and a retransmission message sent following failure of a node of the multi-node data storage system, wherein the write request message is received by the second data storage system from the first data storage system and comprises an indicator of the message type, the data to be written and the region in the data volume where the data is to be written;
determining whether there is an outstanding write request to the second data storage system that corresponds to the region of the data volume in the received message;
wherein, if there is an outstanding write request to the first data storage system that corresponds to the region of the data volume in the received message, the method further comprises:
determining whether the write of the received write request message is the same type as the outstanding write request, and
if the write of the received write request message is the same type as the outstanding write request, writing the data associated with a predetermined one of the first and second data storage systems to the region in the data volume of the second data storage system last, or
if the write of the received write request message is not the same type as the outstanding write request, writing the data associated with the retransmission message to the region in the data volume of the second data storage system after writing the data associated with the resynchronization message.

2. The method of claim 1 , wherein:
if the failed node is the owner node of the second data storage system, the method comprises:
sending a retransmission message of the outstanding write request to the backup node of the second data storage system, the retransmission message comprising an indicator that the write request is a retransmission, the data to be written and the region in the data volume where the data is to be written.

3. The method of claim 1 , further comprising:
in response to a retransmission message, receiving a replication write completion message from the second data storage system, and
sending a host write completion message to the host of the first data storage system.

4. The method of claim 1 , further comprising:
in response to a resynchronization message, receiving a replication write completion message from the second data storage system.

5. The method of claim 1 , wherein:
if the failed node is the backup node of the first data storage system, the method comprises:
determining whether the outstanding write request corresponds to a host write to the owner node of the first data storage system, and if the outstanding write request corresponds to a host write to the owner node of the first data storage system, sending a retransmission message of the outstanding write request to a node of the second data storage system, the retransmission message comprising an indicator that the write request is a retransmission, the data to be written and the region in the data volume where the data is to be written;

if the failed node is the backup node of the second data storage system, the method comprises:
sending a retransmission message of the outstanding write request to the owner node of the second data storage system, the retransmission message comprising an indicator that the write request is a retransmission, the data to be written and the region in the data volume where the data is to be written.

6. The method of claim 1 , wherein:
if the failed node is the owner node of the first data storage system, the method is performed by the backup node of the first data storage system, and
if the failed node is the owner node of the second data storage system or the backup node of the first or second data storage system, the method is performed by the owner node of the first data storage system.

7. The method of claim 1 , wherein the predetermined one of the first and second data storage systems is designated as the leader.

8. The method of claim 1 , wherein:
if there is no outstanding write request to the second data storage system that corresponds to the region of the data volume in the received message, the method further comprises:
writing the data associated with the received write request message to the region of the data volume in the second data storage system.

9. The method of claim 1 , further comprising:
sending a replication write completion message to the first data storage system.

10. A multi-node data storage system, comprising:
a first data storage system having an owner node and a backup node in a first location, and
a second data storage system having an owner node and a backup node in a second location,
wherein the first data storage system is communicatively coupled to the second data storage system and each of the first and second data storage systems includes a copy of the same data volume;
wherein a node of the multi-node data storage system is configured to:
detect failure of a node of the multi-node data storage system;
identify an outstanding write request to the first data storage system;
determine whether the outstanding write request corresponds to a host write to the backup node of the first data storage system;
wherein if the failed node is the owner node of the first data storage system, the node is configured to:
send a retransmission message of the outstanding write request to a node of the second data storage system, when the outstanding write request corresponds to a host write to the backup node of the first data storage system, the retransmission message comprising an indicator that the write request is a retransmission, the data to be written and the region in the data volume where the data is to be written;
read the data region associated with the outstanding write request from the first data storage system and send a resynchronization message to a nod

Assignees

• IBM

Inventors

• Rostagni Florent
• Sipka Andrea
• Wilkinson John

Classifications

• G06F2201/82

  Solving problems relating to consistency · CPC title

• G06F11/1448

  Management of the data involved in backup or backup restore · CPC title

• G06F11/1458

  Management of the backup or restore process · CPC title

• G06F12/00

  Accessing, addressing or allocating within memory systems or architectures (digital input from, or digital output to record carriers, e.g. to disk storage units, G06F3/06) · CPC title

• G06F9/5072 Primary

  Grid computing · CPC title