Cloud migration of file system data hierarchies

Patent metadata

Field	Value
Publication number	US-10558699-B2
Application number	US-201715610456-A
Country	US
Kind code	B2
Filing date	May 31, 2017
Priority date	Jan 6, 2017
Publication date	Feb 11, 2020
Grant date	Feb 11, 2020

Abstract

Techniques described herein relate to systems and methods of data storage, and more particularly to providing layering of file system functionality on an object interface. In certain embodiments, file system functionality may be layered on cloud object interfaces to provide cloud-based storage while allowing for functionality expected from a legacy applications. For instance, POSIX interfaces and semantics may be layered on cloud-based storage, while providing access to data in a manner consistent with file-based access with data organization in name hierarchies. Various embodiments also may provide for memory mapping of data so that memory map changes are reflected in persistent storage while ensuring consistency between memory map changes and writes. For example, by transforming a ZFS file system disk-based storage into ZFS cloud-based storage, the ZFS file system gains the elastic nature of cloud storage.

First claim

What is claimed:

1. A method comprising:
receiving, by one or more virtual devices, a first set of data objects stored in a first data object store, the first set of data objects corresponding to a first plurality of logical blocks, the first plurality of logical blocks corresponding to a first instance of a tree hierarchy of an object-based file system, the first data object store configured to process a first set of attributes of at least one data object of the first set of data objects, wherein the first instance of the tree hierarchy comprises:
(a) a first set of logical data blocks corresponding to file data of one or more files represented by the object-based file system, the first set of logical data blocks including a first logical data block storing data contents of a first file of the object-based file system; and
(b) a first set of logical metadata blocks storing metadata attributes of the one or more files represented by the object-based file system, the first set of logical metadata blocks including a first logical metadata block storing one or more metadata attributes of the first file of the object-based file system, wherein the first set of logical data blocks and the first set of logical metadata blocks are stored in separate data objects within the first set of data objects;

determining, by the one or more virtual devices, that a second data object store does not include one or more data objects corresponding to the first instance of the tree hierarchy, wherein the second data object store is configured to process a second set of attributes for at least one of the one or more data objects; and
in response to determining that the second data object store does not include the one or more data objects:
generating, by the one or more virtual devices, a second set of data objects corresponding to a second plurality of logical blocks, the second plurality of logical blocks corresponding to a second instance of the tree hierarchy, wherein the second instance of the tree hierarchy comprises:
(a) a second set of logical data blocks corresponding to the file data of the one or more files represented by the object-based file system, the second set of logical data blocks including a second logical data block storing data contents of the first file of the object-based file system; and
(b) a second set of logical metadata blocks storing metadata attributes of the one or more files represented by the object-based file system, the second set of logical metadata blocks including a second logical metadata block storing the metadata attributes of the first file of the object-based file system, wherein the second set of logical data blocks and the second set of logical metadata blocks are stored in separate data objects within the second set of data objects; and

storing, by the one or more virtual devices, the second set of data objects corresponding to the second instance of the tree hierarchy, within the second data object store.

2. The method of claim 1 , wherein the first plurality of logical blocks corresponding to the first instance of the tree hierarchy is identical to the second plurality of logical blocks corresponding to the second instance of the tree hierarchy.

3. The method of claim 1 , wherein receiving the first set of data objects stored in the first data object store comprises retrieving, via a first cloud interface device, the first set of data objects as cloud storage objects stored in a first cloud object store, and
wherein storing the second set of data objects stored in the second data object store comprises storing, via a second cloud interface device, a second set of cloud storage objects in a second cloud object store.

4. The method of claim 3 , wherein the first set of attributes processed by the first cloud object store is different from the second set of attributes processed by the second cloud object store.

5. The method of claim 1 , wherein receiving the first set of data objects stored in the first data object store comprises retrieving, via a first cloud interface device, the first set of data objects as cloud storage objects stored in a first cloud object store, and
wherein storing the second set of data objects stored in the second data object store comprises storing the second set of data objects within a local data object store.

6. The method of claim 1 , wherein receiving the first set of data objects corresponding to a first instance of a tree hierarchy comprises:
staging a first subset of the first set of data objects storing the first set of logical data blocks, and staging a second subset of the first set of data objects storing the first set of logical metadata blocks, wherein different staging criteria are applied to the first and second subsets of data objects.

7. A system comprising:
a processing unit comprising one or more processors; and
memory coupled with and readable by the processing unit and storing therein a set of instructions which, when executed by the processing unit, causes the system to:
receive, at one or more virtual devices of the system, a first set of data objects stored in a first data object store, the first set of data objects corresponding to a first plurality of logical blocks, the first plurality of logical blocks corresponding to a first instance of a tree hierarchy of an object-based file system, the first data object store configured to process a first set of attributes of at least one data object of the first set of data objects, wherein the first instance of the tree hierarchy comprises:
(a) a first set of logical data blocks corresponding to file data of one or more files represented by the object-based file system, the first set of logical data blocks including a first logical data block storing data contents of a first file of the object-based file system; and
(b) a first set of logical metadata blocks storing metadata attributes of the one or more files represented by the object-based file system, the first set of logical metadata blocks including a first logical metadata block storing one or more metadata attributes of the first file of the object-based file system, wherein the first set of logical data blocks and the first set of logical metadata blocks are stored in separate data objects within the first set of data objects;

determine, by the one or more virtual devices, that a second data object store does not include one or more data objects corresponding to the first instance of the tree hierarchy, wherein the second data object store is configured to process a second set of attributes for at least one of the one or more data objects; and
in response to determining that the second data object store does not include the one or more data objects:
generate, by the one or more virtual devices, a second set of data objects corresponding to a second plurality of logical blocks, the second plurality of logical blocks corresponding to a second instance of the tree hierarchy, wherein the second instance of the tree hierarchy comprises:
(a) a second set of logical data blocks corresponding to the file data of the one or more files represented by the object-based file system, the second set of logical data blocks including a second logical data block storing data contents of the first file of the object-based file system; and
(b) a second set of logical metadata blocks storing metadata attributes of the one or more files represented by the object-based file system, the second set of logical metadata blocks including a second logical metadata block storing the metadata attributes of the first file of the object-based file system, wherein the second set of logical data blocks and the second set of logical metadata blocks are stored in separate data objects within the second set of data objects;

Assignees

• Oracle Int Corp

Inventors

• Maybee Mark
• Kremer James
• Ledru Pascal

Classifications

• G06F2009/4557

  Distribution of virtual machine instances; Migration and load balancing · CPC title

• G06F2009/45595

  Network integration; Enabling network access in virtual machine instances · CPC title

• G06F9/5077

  Logical partitioning of resources; Management or configuration of virtualized resources (specific details on emulation or internal functioning of virtual machines G06F9/455) · CPC title

• G06F11/1464

  for networked environments · CPC title

• G06F2212/313

  In storage device · CPC title