Distributed failover of a back-end storage director

Patent metadata

Field	Value
Publication number	US-11467930-B2
Application number	US-201916567026-A
Country	US
Kind code	B2
Filing date	Sep 11, 2019
Priority date	Sep 11, 2019
Publication date	Oct 11, 2022
Grant date	Oct 11, 2022

Abstract

Distributed failover of a failed BE, and failback of the failed BE when it recovers, are described. A separate pool of memory may be reserved for each BE on a storage system. Each such pool may be further divided into sub-pools. When a BE fails, the one or more other active BEs on the storage system may be notified, and each such active BE may attempt to take ownership of one or more sub-pools of the BE and execute the one or more pending write operations of the failed BE. Upon completing execution of the pending write operations in a failed-over sub-pool, the ownership of the sub-pool may be returned to the failed BE. When the failed BE recovers, the recovered BE may attempt to take back control of one or more of its sub-pools.

First claim

What is claimed is:

1. For a storage system including a plurality of physical storage devices, a plurality of directors processing I/O operations for the plurality of physical storage devices, and a memory including a plurality of memory pools, each of the plurality of memory pools corresponding to a respective one of the plurality of directors and including a plurality of sub-pools for use by the respective director in the processing of I/O operations, a method comprising:
receiving an indication that a first of the plurality of directors has failed, wherein the plurality of directors denotes all back-end directors in the storage system configured to access the plurality of physical storage devices;
performing processing by all active, non-failed back-end directors of the storage system to attempt to take ownership of at least one of the plurality of sub-pools of a first memory pool of the plurality of memory pools corresponding to the first director that failed, wherein said processing includes determining whether the at least one sub-pool of the plurality of sub-pools of the first memory pool corresponding to the first director includes pending write operations;
responsive to said processing by a second of the plurality of directors, assigning ownership of the at least one sub-pool to the second director; and
the second director processing the pending write operations of the at least one sub-pool.

2. The method of claim 1 , further comprising:
assigning ownership of at least one other sub-pool of the first memory pool to a third of the plurality of directors.

3. The method of claim 1 , wherein the step of determining includes the second director determining, for at least two of the plurality of sub-pools of the first memory pool:
whether another of the plurality of directors is assigned ownership of the each of the at least two sub-pools; and
if no other of the plurality of directors is assigned ownership of each of the at least two sub-pools, assigning ownership of the at least two sub-pools to the second director.

4. The method of claim 1 , wherein the step of determining whether the at least one of the plurality of sub-pools of the first memory pool includes pending write operations is performed separately by each active, non-failed back-end director of the storage system.

5. The method of claim 1 , wherein the step of determining includes determining that at least two of the plurality of sub-pools of the first memory pool include pending write operations, and wherein the method further includes:
maintaining ownership of at least one of the at least two sub-pools with the first director.

6. The method of claim 1 , further comprising:
the second director completing processing of the pending write operations of the at least one sub-pool; and
in response to the completion of the processing, re-assigning ownership of the at least one sub-pool back to the first director.

7. The method of claim 1 , further comprising:
the first director becoming active again after having failed;
determining which of the plurality of sub-pools of the first memory pool have ownership currently assigned to another of the plurality of directors; and
for each sub-pool of the first memory pool determined to have ownership currently assigned to another of the plurality of directors, performing first processing including:
determining whether the other director having ownership is active or failed,
if the other director is failed, re-assigning ownership of the sub-pool back to the first director, and
if the other director is active, maintaining ownership of the sub-pool with the other director until completion by the other director of any pending write operations of the sub-pool.

8. The method of claim 1 , wherein each of the plurality of directors are communicatively coupled by a fabric to each of the plurality of physical storage devices.

9. A storage system comprising:
a plurality of physical storage devices;
a plurality of directors processing I/O operations with the plurality of physical storage devices, wherein the plurality of directors denotes all back-end directors in the storage system configured to access the plurality of physical storage devices;
a memory including a plurality of memory pools, each of the plurality of memory pools corresponding to a respective one of the plurality of directors and including a plurality of sub-pools for use by the respective director in the processing of I/O operations,
wherein the memory has code stored thereon that, when executed, performs a method including:
receiving an indication that a first of the plurality of directors has failed;
performing processing by all active, non-failed back-end directors of the storage system to attempt to take ownership of at least one of the plurality of sub-pools of a first memory pool of the plurality of memory pools corresponding to the first director that failed, wherein said processing includes determining whether the at least one sub-pool of the plurality of sub-pools of the first memory pool corresponding to the first director includes pending write operations;
responsive to said processing by a second of the plurality of directors, assigning ownership of the at least one sub-pool to the second director; and
the second director processing the pending write operations of the at least one sub-pool.

10. The storage system of claim 9 , wherein the method further includes:
assigning ownership of at least one other sub-pool of the first memory pool to a third of the plurality of directors.

11. The storage system of claim 9 , wherein the step of determining includes the second director determining, for at least two of the plurality of sub-pools of the first memory pool:
whether another of the plurality of directors is assigned ownership of the each of the at least two sub-pools; and
if no other of the plurality of directors is assigned ownership of each of the at least two the sub-pools, assigning ownership of the at least two sub-pools to the second director.

12. The storage system of claim 9 , wherein the step of determining whether the at least one of the plurality of sub-pools of the first memory pool includes pending write operations is performed separately by each active, non-failed back-end director of the storage system.

13. The storage system of claim 9 , wherein the step of determining includes determining that at least two of the plurality of sub-pools of the first memory pool include pending write operations, and wherein the method further includes:
maintaining ownership of at least one of the at least two sub-pools with the first director.

14. The storage system of claim 9 , wherein the method further includes:
the second director completing processing of the pending write operations of the at least one sub-pool; and
in response to the completion of the processing, re-assigning ownership of the at least one sub-pool back to the first director.

15. The storage system of claim 9 , wherein the method further includes:
the first director becoming active again after having failed;
determining which of the plurality of sub-pools of the first memory pool have ownership currently assigned to another of the plurality of directors; and
for each sub-pool of the first memory pool determined to have ownership currently assigned to another of the plurality of directors, performing first processing including:
determining whether the other director having ownership is active or failed,
if the other director is failed, re-assigning ownership of the sub-pool back to the first director, and
if the other director is ac

Assignees

• Emc Ip Holding Co Llc

Inventors

• Yin Peng
• Li Ningdong
• Wang Jiahui
• Hu Shao Kuang

Classifications

• G06F11/2092 Primary

  Techniques of failing over between control units · CPC title

• G06F3/0683

  Plurality of storage devices · CPC title

• G06F11/2048

  where the redundant components share neither address space nor persistent storage · CPC title

• G06F3/0604

  Improving or facilitating administration, e.g. storage management · CPC title

• G06F3/0659

  Command handling arrangements, e.g. command buffers, queues, command scheduling · CPC title