Guaranteed file system hierarchy data integrity in cloud object stores

Patent metadata

Field	Value
Publication number	US-2018196818-A1
Application number	US-201715610444-A
Country	US
Kind code	A1
Filing date	May 31, 2017
Priority date	Jan 6, 2017
Publication date	Jul 12, 2018
Grant date	—

Abstract

Techniques described herein relate to systems and methods of data storage, and more particularly to providing layering of file system functionality on an object interface. In certain embodiments, file system functionality may be layered on cloud object interfaces to provide cloud-based storage while allowing for functionality expected from a legacy applications. For instance, POSIX interfaces and semantics may be layered on cloud-based storage, while providing access to data in a manner consistent with file-based access with data organization in name hierarchies. Various embodiments also may provide for memory mapping of data so that memory map changes are reflected in persistent storage while ensuring consistency between memory map changes and writes. For example, by transforming a ZFS file system disk-based storage into ZFS cloud-based storage, the ZFS file system gains the elastic nature of cloud storage.

First claim

What is claimed:

1 . A method comprising:
receiving, from an application layer of a ZFS system and through a system call interface of an interface layer of the ZFS system, a file; creating, via one or more virtual devices of a plurality of virtual devices at a virtual device layer of the ZFS system, data blocks that correspond to the file; causing, by a cloud interface appliance, storage of cloud storage objects in a cloud object store, the cloud storage objects comprising data of the data blocks and corresponding metadata of according to a tree hierarchy, wherein:
a first subset of the cloud storage objects comprises the data of the data blocks, the first subset corresponding to leaf nodes of the tree hierarchy;
a second subset of the cloud storage objects comprises the corresponding metadata, the second subset corresponding to non-leaf nodes of the tree hierarchy; and
the metadata comprises respective checksums generated for each cloud storage object of the first subset of cloud storage objects;
receiving, from the application layer of the ZFS system, a request to perform a transaction with respect to the file; translating, by a data management unit of a transactional object layer of the ZFS system, the request into an I/O request; and based at least in part on the I/O request, performing, by a cloud interface appliance, a write operation to modify the tree hierarchy at least in part by causing storage of an additional cloud storage objects to result in a modified tree hierarchy, wherein:
a third subset of the additional cloud storage objects comprises additional data, the third subset corresponding to one or more additional leaf nodes as modifications to the tree hierarchy;
a fourth subset of the additional cloud storage objects comprises additional metadata corresponding to the additional data, the fourth subset corresponding to additional non-leaf nodes of the tree hierarchy; and
the additional metadata comprises additional respective checksums generated for each cloud storage object of the third subset of the additional cloud storage objects.

2 . The method of claim 1 , further comprising:
receiving, from the application layer of the ZFS system, a subsequent request to perform a subsequent transaction with respect to the file; translating, by the data management unit, the subsequent request into a subsequent I/O request; and based at least in part on the subsequent I/O request, performing, by the cloud interface appliance, a read operation with respect to the modified tree hierarchy, wherein the read operation comprises receiving a version of at least one cloud storage object from the cloud object store; and checking the version of the at least one cloud storage object with a reference checksum.

3 . The method of claim 2 , wherein the checking the version of the at least one cloud storage object with the reference checksum comprises:
processing, by a storage pool allocator or the cloud interface appliance, the version of the at least one cloud storage object to generate a generated checksum; retrieving, by a storage pool allocator or the cloud interface appliance, specific metadata that is stored separately from the at least one cloud storage object and that corresponds to a specific non-leaf node of the non-leaf nodes or additional non-leaf nodes according to the modified tree hierarchy, wherein the specific non-leaf is a parent node with respect to the at least one cloud storage object; identifying, by the storage pool allocator or the cloud interface appliance, from the specific metadata the reference checksum for the at least one cloud storage object, the reference checksum being associated with a last modified version of the cloud storage object; and determining whether the generated checksum matches the reference checksum.

4 . The method of claim 3 , wherein the performing, by the cloud interface appliance, the write operation to modify the tree hierarchy is performed according to a copy-on-write process.

5 . The method of claim 4 , further comprising:
causing, via the cloud interface appliance or another cloud interface appliance, storage of the cloud storage objects and the additional cloud storage objects in second cloud object store so that a second cloud instance of the modified tree hierarchy is stored in the second cloud object store, wherein the modified tree hierarchy stored in the cloud object store corresponds to a first cloud instance of the modified tree hierarchy; wherein the causing storage of the cloud storage objects and the additional cloud storage objects in second cloud object store comprises independent generating additional checksums for each cloud storage object stored as leaf nodes in the second cloud object store.

6 . The method of claim 5 , further comprising:
after the storage of the cloud storage objects and the additional cloud storage objects in the second cloud object store, identifying an unsynchronized state of the first cloud instance of the tree hierarchy with respect to the second cloud instance of the tree hierarchy based at least in part on at least one checksum stored in at least one non-leaf node of the first cloud instance and at least one checksum stored in at least one non-leaf node of the second cloud instance.

7 . The method of claim 6 , further comprising:
in response to identifying the unsynchronized state, initiating, by the ZFS system, synchronization of the first cloud instance and the second cloud instance based at least in part on one or more remediation processes.

8 . A system comprising:
one or more processors communicatively coupled to memory, the one or more processors to facilitate:
receiving, from an application layer of a ZFS system and through a system call interface of an interface layer of the ZFS system, a file;
creating, via one or more virtual devices of a plurality of virtual devices at a virtual device layer of the ZFS system, data blocks that correspond to the file;
causing, by a cloud interface appliance, storage of cloud storage objects in a cloud object store, the cloud storage objects comprising data of the data blocks and corresponding metadata of according to a tree hierarchy, wherein:
a first subset of the cloud storage objects comprises the data of the data blocks, the first subset corresponding to leaf nodes of the tree hierarchy;
a second subset of the cloud storage objects comprises the corresponding metadata, the second subset corresponding to non-leaf nodes of the tree hierarchy; and
the metadata comprises respective checksums generated for each cloud storage object of the first subset of cloud storage objects;

receiving, from the application layer of the ZFS system, a request to perform a transaction with respect to the file;
translating, by a data management unit of a transactional object layer of the ZFS system, the request into an I/O request; and
based at least in part on the I/O request, performing, by a cloud interface appliance, a write operation to modify the tree hierarchy at least in part by causing storage of an additional cloud storage objects to result in a modified tree hierarchy, wherein:
a third subset of the additional cloud storage objects comprises additional data, the third subset corresponding to one or more additional leaf nodes as modifications to the tree hierarchy;
a fourth subset of the additional cloud storage objects comprises additional metadata corresponding to the additional data, the fourth subset corresponding to additional non-leaf nodes of the tree hierarchy; and
the additional metadata comprises additional respective checksums generated for each cloud storage object of the third subset of the additional cloud storage objects.

9 . The system o

Assignees

• Oracle Int Corp

Inventors

• Maybee Mark
• Kremer James
• Latushkin Victor
• Gureja Ankit
• Dunlop Kenneth

Classifications

• H04L63/0428

  wherein the data content is protected, e.g. by encrypting or encapsulating the payload · CPC title

• G06F12/123

  with age lists, e.g. queue, most recently used [MRU] list or least recently used [LRU] list · CPC title

• G06F11/3037

  where the computing system component is a memory, e.g. virtual memory, cache (accessing, addressing or allocating within memory systems or architectures G06F12/00; checking stores for correct operation G11C29/00) · CPC title

• G06F2201/81

  Threshold · CPC title

• G06F16/2365

  Ensuring data consistency and integrity · CPC title