Systems, devices and methods for managing file system replication

Patent metadata

Field	Value
Publication number	US-10725974-B2
Application number	US-201615358936-A
Country	US
Kind code	B2
Filing date	Nov 22, 2016
Priority date	Nov 22, 2016
Publication date	Jul 28, 2020
Grant date	Jul 28, 2020

Abstract

A method for managing file system replication can include: collecting event data associated with file system events, each of the file system events indicative of a modification of at least one file at a first file system location; for each of the file system events, generating at least one file system action to apply the modification of the at least one file at the first file system location to at least one corresponding file at a second file system location; storing the at least one generated file system action in an action buffer; identifying dependencies between file system actions in the action buffer; and scheduling the file system actions in the action buffer for execution based on the dependencies.

First claim

What is claimed is:

1. A method for managing replication of files of distinct file systems, each file system at a different location and comprising files distributed over at least one machine at the different location, the method comprising:
receiving, by a replicator device between a first file and a second file system of the distinct file systems, event data associated with file system events, each file system event of the file system events related to a modification of at least one file located at the first file system of the distinct file systems;
for the each file system event of the file system events:
generating, by the replicator device, at least one file system action to be executed at the second file system of the distinct file systems to replicate the modification of the at least one file located at the first file system, wherein the at least one file system action causes the modification of the at least one file located at the first file system to be applied to a replica of the at least one file located at the second file system; and
storing, by the replicator device, the at least one file system action for the file system event in an action buffer;

determining, by the replicator device, that a number of file system actions in the action buffer exceeds a size threshold number in a batch optimization trigger;
in response to the determining:
identifying, by the replicator device, dependencies between the file system actions in the action buffer that are rename events or unlink events, wherein a rename event indicates that a file located at the first file system has been renamed and an unlink event indicates that the file located at the first file system has been deleted;
organizing, by the replicator device, the file system actions stored in the action buffer without violating the identified dependencies between file system actions;
scheduling, by the replicator device, the file system actions stored in the action buffer for execution based on the dependencies; and
communicating, by the replicator device, the scheduled file system actions to the second file system for execution at the second file system.

2. The method of claim 1 , wherein scheduling the file system actions in the action buffer for execution comprises:
upon determining that all dependencies for a particular file system action in the action buffer have been resolved, identifying the particular file system action as ready for execution; and
scheduling the particular file system action in the action buffer identified as ready for execution.

3. The method of claim 1 , comprising:
receiving an execution response message regarding an execution or attempted execution of a scheduled file system action; and
updating the action buffer based on the received execution response message.

4. The method of claim 3 , wherein updating the action buffer based on the received execution response message comprises:
when the received execution response message indicates that execution of one of the scheduled file system actions was successful, resolving a dependency for at least one file system action in the action buffer that depended on one scheduled file system action that was successful.

5. The method of claim 1 , comprising:
upon determining one of the file system events associated with the received event data indicates that content data of at least one file located at the first file system is being modified:
monitoring a length of the content data being modified; and
generating a copy chunk action when the length of the content data being modified indicates that a new chunk of modified content data having a defined chunk size is available for copying, the copy chunk action to replicate the new chunk of modified content data located at the second file system.

6. The method of claim 1 , comprising:
upon determining one of the file system events associated with the received event data indicates that content data of at least one file located at the first file system is being modified:
generating an initial copy chunk action to replicate an initially available portion of the modified content data at the first file system; and
generating a subsequent copy chunk action to replicate a next available portion of the modified content data when an execution response message is received indicating that a previous copy chunk action to replicate the content data was successful.

7. The method of claim 1 , wherein generated actions comprise file location information for the at least one file located at the first file system.

8. The method of claim 1 , wherein organizing comprises reordering one or more file system actions in the action buffer.

9. The method of claim 1 , wherein organizing comprises combining or eliminating one or more related actions stored in the action buffer, wherein two or more actions are related when at least one file is involved in each of the two or more actions.

10. The method of claim 9 , comprising: delaying the combining or eliminating of the one or more related actions until the batch optimization trigger is detected.

11. The method of claim 1 ,
wherein the event data includes first modification event data associated with a first modification event for modifying a first file at the first file system,
wherein the file system actions include a first modification action for modifying a first replica of the first file at the second file system,
wherein the method further comprises receiving a first close event associated with closing the first file at the first file system after the modifying the first file, and
wherein the scheduling comprises scheduling the first modification action such that the first modification action is communicated to the second file system for execution at the second file system before the receiving the first close event.

12. A system for managing replication of files of distinct file systems, each file system at a different location and comprising files distributed over at least one machine at the different location, the system comprising:
at least one processor configured for:
receiving, by the system between a first file system and a second file system of the distinct file systems, event data associated with file system events, each file system event of the file system events related to a modification of at least one file located at the first file system of the distinct file systems;

for the each file system event of the file system events:
generating at least one file system action to be executed at the second file system of the distinct file systems to replicate the modification of the at least one file located at the first file system, wherein the at least one file system action causes the modification of the at least one file located at the first file system to be applied to a replica of the at least one file located at the second file system;
storing the at least one file system action for the file system event in an action buffer;

determining that a number of file system actions in the action buffer exceeds a size threshold number in a batch optimization trigger;
in response to the determining:
identifying dependencies between the file system actions in the action buffer that are rename events or unlink events, wherein a rename event indicates that a file located at the first file system has been renamed and an unlink event indicates that the file located at the first file system has been deleted;
organizing the file system actions stored in the action buffer without violating the identified dependencies between file system actions;
scheduling the file system actions in the action buffer based on the dependencies; and
communica

Assignees

• Huawei Tech Co Ltd

Inventors

• Chen Chen
• CHEN Jin
• Ke Xiaodi
• Guo Lei
• Chen Chong

Classifications

• G06F16/122

  using management policies (point-in-time backing up or restoration of persistent data G06F11/1446; file migration policies for HSM systems G06F16/185) · CPC title

• G06F16/178 Primary

  Techniques for file synchronisation in file systems · CPC title