Unordered idempotent logical replication operations

Patent metadata

Field	Value
Publication number	US-9659026-B2
Application number	US-201314145728-A
Country	US
Kind code	B2
Filing date	Dec 31, 2013
Priority date	Apr 30, 2009
Publication date	May 23, 2017
Grant date	May 23, 2017

Abstract

A method and apparatus for replicating a data container of a source storage server at the logical level in an unordered stream of individual data units are described. In certain embodiments, the replication operations can be performed without regard to the differences in geometry of physical persistent storage of the source and destination storage servers and without regard to the differences in data container format of the source and destination storage servers. An initial replication operation of the source data container is performed by transferring the data objects within the source data container to a replica data container in the destination storage server as an unordered stream of individual data units while preserving data object identifiers of the source data objects. Afterwards, incremental replication operations can be performed to capture modifications to the source data container over time after the initial replication operation.

First claim

What is claimed is:

1. A method, comprising:
creating a current point-in-time image of a file system stored within a first server, the file system configured to store multiple data objects, each data object having assigned to it a corresponding data object identifier;
scanning data objects within the file system to identify changes in the file system that occurred after an earlier replication operation was performed, by comparing the current point-in-time image with a previous point-in-time image corresponding to the earlier replication operation, the changes in the file system including updates to a data block of a file having both modified data blocks and unmodified data blocks and a generation number of the file; and
transferring, in an unordered stream to a second server, the changes in the file system as multiple individual data units while preserving corresponding file system object identifiers to replicate the file system at the second server; wherein the transferring includes, for the file, transferring only the modified data blocks, and for a directory, transferring only modified directory entries, wherein the file is transferred before the directory within which the file is located.

2. The method of claim 1 , wherein the transferring comprises:
transferring the changes in the file system as multiple individual data units in the unordered stream to the second server without regard to a hierarchical structure of the file system, wherein the first server and the second server are storage servers.

3. The method of claim 1 , wherein the transferring comprises:
transferring the changes in the file system as multiple individual data units in the unordered stream to the second server without regard to differences in geometry of physical persistent storage of the first server and the second server and without regard to differences in file system format of the first server and the second server.

4. The method of claim 1 , further comprising:
scanning file system objects within the file system by examining inode entries in an inode file of the file system from beginning to end in a single pass through the inode file of the file system, wherein when an inode is the directory, scanning individual directory entries; and
transferring the file system objects to the second server without first determining where the file system objects are located in a directory tree of the file system.

5. The method of claim 1 , wherein a file system object identifier is an inode number of an inode entry in an inode file of the first server.

6. The method of claim 5 , wherein the inode file is a data structure representing a master list of file system objects and each inode entry of the inode file identifies a particular file system object within the file system.

7. The method of claim 6 , further comprising:
initiating a data operation when the file system object to be replicated is the file; and
initiating a directory operation when the file system object to be replicated is the directory.

8. The method of claim 7 , wherein the data operation includes:
a file block number of the data block being replicated;
a corresponding inode number for the data block corresponding to the file block number;
a block of data corresponding to the file block number; and
the generation number of the file.

9. The method of claim 7 , wherein the directory operation includes:
an inode number of the directory corresponding to a directory entry being replicated;
a generation number of the directory;
a name of the file system object referenced by the directory entry;
an inode number of the file system object referenced by the directory entry; and
a generation number of the file system object referenced by the directory entry.

10. The method of claim 7 , further comprising:
transferring data blocks of the file of the file system to the second server before transferring metadata describing the file.

11. A computer-readable storage medium storing computer-executable instructions that when executed cause performance of operations, the operations comprising:
receiving, in an unordered stream from a first server at a second server, file system objects, the file system objects in the unordered stream having changes that occurred in a file system of the first server after a previous replication operation was performed, wherein each of the file system objects includes a corresponding file system object identifier, the changes in the file system of the first server including updates to a directory entry of a directory having modified directory entries and unmodified directory entries and a generation number; wherein the receiving includes, for a file, receiving only modified data blocks, and for the directory, receiving only the modified directory entries and the generation number, wherein the modified data blocks of the file are transferred to the second server before the directory within which the file is located;
replicating the file system objects at the second server using the file system object identifiers; and
creating a current point-in-time image of a replicated file system of the second server once the file system objects are replicated at the second server so that corresponding point-in-time images of the first server and second server are identical.

12. The computer-readable storage medium of claim 11 , wherein each of the file system object identifiers is an inode number of an inode entry in an inode file of the first server, and wherein the inode file is a data structure representing a master list of file system objects and each inode entry identifies a particular file system object within the file system.

13. The computer-readable storage medium of claim 12 , further comprising:
matching the inode number of the file system object in the inode file with a corresponding inode number in a replicated inode file at the second server; and
adding the file system object to an inode entry in the replicated inode file at the second server so that the inode number of the inode file is preserved in the replicated inode file of the second server.

14. The computer-readable storage medium of claim 13 , further comprising:
storing data associated with each file system object to an unused block of data in physical persistent storage of the second server; and
updating a buffer tree representing a logical block structure of the second file system to reference the data of each file system object, wherein the buffer tree includes a hierarchical metadata structure comprising pointers to logical blocks of data in the second file system.

15. The computer-readable storage medium of claim 14 , wherein the updating the buffer tree comprises:
updating an indirect block in the buffer tree to include a pointer to the data of each file system object;
storing each indirect block to a different unused block of data in physical persistent storage of the second server; and
repeating the updating and storing until all pointers to each data of the file system object in the buffer tree have been updated.

16. The computer-readable storage medium of claim 15 , further comprising:
assigning the file block number to a highest-level indirect block in the buffer tree referencing the data of the file system object, wherein the first server and the second server are storage servers.

17. A computing device for replicating data in a file system, comprising:
a processor; and
memory comprising processor-executable instructions that when executed by the processor cause performance of operations, the operations comprising:

Assignees

• Netapp Inc

Inventors

• Leverett Bruce
• Kazar Michael
• Driscoll Alan Stuart
• Corbett Peter Frank
• Shah Devang K
• Simharajan Srishylam
• Smith Toby

Classifications

• G06F16/184

  implemented as replicated file system · CPC title

• G06F3/065

  Replication mechanisms · CPC title

• G06F16/178 Primary

  Techniques for file synchronisation in file systems · CPC title

• G06F16/128

  Details of file system snapshots on the file-level, e.g. snapshot creation, administration, deletion (error detection or correction of the data by redundancy in operations or in hardware G06F11/14, G06F11/16) · CPC title

• G06F17/30088

  Physics · mapped topic