Data node fencing in a distributed file system

Patent metadata

Field	Value
Publication number	US-9753954-B2
Application number	US-201314024585-A
Country	US
Kind code	B2
Filing date	Sep 11, 2013
Priority date	Sep 14, 2012
Publication date	Sep 5, 2017
Grant date	Sep 5, 2017

Abstract

Systems and methods for data node fencing in a distributed file system to prevent data inconsistencies and corruptions are disclosed. An embodiment includes implementing a protocol whereby data nodes detect a failover and determine an active name node based on transaction identifiers associated with transaction requests. The data nodes also provide to the active name node block location information and an acknowledgment. The embodiment further includes a protocol whereby a name node refrains from issuing invalidation requests to the data nodes until the name node receives acknowledgments from all data nodes that are functional.

First claim

What is claimed is:

1. A method for maintaining data correctness in a Hadoop™ based distributed cluster during a failover, in which an original name node is switched to a backup name node due to failure of the original name node, the distributed cluster having a plurality of data nodes and one or more processors, the method being performed by the one or more processors and comprising:
on the backup name node:
assuming an active role to become a new active name node, upon detecting that the original name node has failed;
flagging all of the plurality of data nodes as untrusted;
for each data node among the plurality of data nodes:
queuing, instead of issuing, commands intended for a data node until the data node is flagged as trusted, and
upon receiving an acknowledgement from the data node acknowledging the assumption of the active role of the backup name node, flagging the data node as trusted; and

on a respective data node:
receiving a first command with a first transaction number from a first name node;
receiving a second command with a second transaction number from a second name node, wherein the second transaction number is greater than the first transaction number; and
sending an acknowledgment of an active role to the second name node.

2. The method of claim 1 , further comprising:
sending a message to the data node, wherein the message includes a most recent transaction identifier known to the backup name node assuming the active role.

3. The method of claim 1 , wherein commands on any block with replicated data on untrusted data nodes are queued.

4. The method of claim 1 , further comprising receiving a data report in addition to the acknowledgment of the active role from the data node.

5. The method of claim 4 , wherein the data report includes information regarding location of replicated data stored in the data node.

6. The method of claim 4 , wherein each data report includes a list of pending deletions.

7. A Hadoop™ based distributed cluster comprising an original name node, a backup name node, and a distributed file system having a plurality of data nodes,
wherein one or more processors of the backup name node are configured to perform:
assuming an active role to become a new active name node, upon detecting that the original name node has failed;
flagging all of the plurality of data nodes as untrusted;
for each data node among the plurality of data nodes:
queuing, instead of issuing, commands intended for a data node until the data node is flagged as trusted; and
upon receiving an acknowledgement from the data node acknowledging the assumption of the active role of the backup name node, flagging the data node as trusted, and

wherein one or more processors of a respective data node are configured to perform;
receiving a first command with a first transaction number from a first name node;
receiving a second command with a second transaction number from a second name node, wherein the second transaction number is greater than the first transaction number; and
sending an acknowledgment of an active role to the second name node.

8. A machine-readable storage medium having stored thereon instructions which, when executed by one or more processors, configure the processors to performs a method in a Hadoop™ based distributed cluster comprising a plurality of name nodes and a plurality of data nodes and having a distributed file system, the method comprising:
on the backup name node:
assuming an active role to become a new active name node, upon detecting that the original name node has failed;
flagging all of the plurality of data nodes as untrusted;
for each data node among the plurality of data nodes;
queuing, instead of issuing, commands intended for a data node until the data node is flagged as trusted, and
upon receiving an acknowledgement from the data node acknowledging the assumption of the active role of the backup name node, flagging the data node as trusted; and

on a respective data node:
receiving a first command with a first transaction number from a first name node;
receiving a second command with a second transaction number from a second name node, wherein the second transaction number is greater than the first transaction number; and
sending an acknowledgment of an active role to the second name node.

9. The cluster of claim 7 , wherein the data nodes are configured to ignore commands from other name nodes that issue commands having a transaction identifier lower than a transaction identifier associated with a command issued by the backup name node.

Assignees

• Cloudera Inc

Inventors

• Lipcon Todd
• Myers Aaron T
• Collins Eli

Classifications

• G06F16/1824

  implemented using Network-attached Storage [NAS] architecture (distributed or networked storage systems G06F3/067; protocols for distributed storage of data in a network H04L67/1097) · CPC title

• G06F11/2046

  where the redundant components share persistent storage (G06F11/2043 takes precedence) · CPC title

• G06F16/215 Primary

  Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors · CPC title

• H04L41/0836

  to enhance reliability, e.g. reduce downtime · CPC title

• G06F11/2038

  with a single idle spare processing component · CPC title