Using log objects in object storage for durability of file objects in volatile memory

Patent metadata

Field	Value
Publication number	US-10901643-B2
Application number	US-201916586770-A
Country	US
Kind code	B2
Filing date	Sep 27, 2019
Priority date	Mar 20, 2017
Publication date	Jan 26, 2021
Grant date	Jan 26, 2021

Abstract

A file system interface receives, from one or more clients, file system operations over a network. In response to receiving a file system operation for a file, a file-object manager calls an object storage interface to store, via an application-level network protocol, a log object indicating the file system operation into a file system log in the object storage system and creates or updates a file object in a file object construction buffer in volatile system memory. To evict the file object, the file-object manager calls the object storage interface to store the file object into a file object store in the object storage system. The file-object manager does not have access to local persistent storage. After power failure, the file-object manager calls the object storage interface to retrieve log objects from the file system log and replays file system operations to restore file objects in the volatile system memory.

First claim

What is claimed is:

1. A system, comprising:
one or more processors; and
one or more memories, wherein the one or more memories have stored thereon instructions, which when executed by the one or more processors to implement a file-object manager, wherein the file-object manager is configured to:
determine that one or more eviction criteria are met to evict a file object in a file object construction buffer in a volatile system memory, wherein the file object corresponds to a file;
in response to the determination that the one or more eviction criteria are met, call an object storage interface to store, via an application-level network protocol, the file object into a file object store in an object storage system; and
in response to a reboot, power failure, or other event that results in loss of content stored in the volatile system memory:
call the object storage interface to retrieve, via the application-level network protocol, log objects from a file system log of the object storage system; and
replay file system operations indicated by the log objects in a same sequential order that the file system operations were previously applied for one or more file objects in the volatile system memory to restore the one or more file objects in the volatile system memory.

2. The system as recited in claim 1 , wherein the file-object manager is further configured to, before the determination that the one or more eviction criteria are met to evict the file object in the file object construction buffer:
in response to receiving by a file system interface a file system operation for the file:
call the object storage interface to store a log object into the file system log in the object storage system, wherein the log object indicates the file system operation for the file; and
create or update, based on the file system operation for the file, the file object in the file object construction buffer in the volatile system memory, wherein the file-object manager does not have access to local persistent storage.

3. The system as recited in claim 1 , wherein, for a file system write received from a client, the file-object manager is configured to:
upon creating or updating a corresponding file object in the file object construction buffer and receiving an acknowledgment from the object storage interface that a corresponding log object is stored in the file system log of the object storage system, instruct the file system interface to acknowledge to the client that the file system write is complete.

4. The system as recited in claim 1 , wherein the one or more eviction criteria comprises at least one of:
available storage space for objects in the file object construction buffer falling below a threshold value, or
the file object being the least recently used file object or being in a group of least recently used file objects in the file object construction buffer, or
receiving an indication to commit the file.

5. The system as recited in claim 1 , wherein the system further comprises a file system interface, wherein the file system interface is configured to:
provide a network file system mount point for one or more clients to access over a network, wherein the one or more file objects correspond to respective files for the network file system.

6. The system as recited in claim 1 , wherein prior to the reboot, power failure, or other event, the one or more file objects exist as sequential data structures in a continuous address range of the volatile system memory.

7. A method, comprising:
performing, by a file-object manager implemented on one or more computing devices:
determining that one or more eviction criteria are met to evict a file object in a file object construction buffer in a volatile system memory, wherein the file object corresponds to a file;
in response to determining that the one or more eviction criteria are met, calling an object storage interface to store, via an application-level network protocol, the file object into a file object store in an object storage system; and
in response to a reboot, power failure, or other event that results in loss of content stored in the volatile system memory:
calling the object storage interface to retrieve, via the application-level network protocol, log objects from a file system log of the object storage system; and
replaying file system operations indicated by the log objects in a same sequential order that the file system operations were previously applied for one or more file objects in the volatile system memory to restore the one or more file objects in the volatile system memory.

8. The method as recited in claim 7 , further comprising performing, by the file-object manager:
before determining that the one or more eviction criteria are met to evict the file object in the file object construction buffer:
in response to receiving by a file system interface a file system operation for the file:
calling the object storage interface to store a log object into the file system log in the object storage system, wherein the log object indicates the file system operation for the file; and
creating or updating, based on the file system operation for the file, the file object in the file object construction buffer in the volatile system memory, wherein the file-object manager does not have access to local persistent storage.

9. The method as recited in claim 7 , further comprising performing, by the file-object manager:
for a file system read received from a client:
determining whether a particular file object corresponding to the file system read exists in the file object construction buffer in the volatile system memory; and
in response to determining that the particular file object corresponding to the file system does not exist in the file object construction buffer in the volatile system memory:
calling the object storage interface to retrieve, via an application-level network protocol, the particular file object from the file object store in the object storage system; and
loading the particular file object into the file object construction buffer in the volatile system memory.

10. The method as recited in claim 7 , wherein a minimum required size of a given log object is smaller than a minimum required size of a given file object.

11. The method as recited in claim 10 , wherein the minimum required size of the given file object corresponds to a minimum required size of file objects for storage in an object storage service of a service provider network.

12. The method as recited in claim 7 , wherein the one or more eviction criteria comprises at least one of:
available storage space for objects in the file object construction buffer falling below a threshold value, or
the file object being the least recently used file object or being in a group of least recently used file objects in the file object construction buffer, or
receiving an indication to commit the file.

13. The method as recited in claim 7 , further comprising performing, by the file-object manager:
for a file system write received from a client:
in response to determining that the file system write is a durable write:
upon creating or updating a corresponding file object in the file object construction buffer and receiving an acknowledgment from the object storage interface that a corresponding log object is stored in the file system log of the object storage system, instructing, by the file-object manager, a file system interface to acknowledge to the client that the file system write is complete.

14. One or more non-transitory computer-accessible storage media storing program

Assignees

• Amazon Tech Inc

Inventors

• Jamail John
• Mesard Wayne
• Wharton Ian Marshal

Classifications

• G06F3/0619

  in relation to data integrity, e.g. data losses, bit errors · CPC title

• G06F3/0656

  Data buffering arrangements · CPC title

• G06F3/067

  Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS] · CPC title

• G06F3/0652

  Erasing, e.g. deleting, data cleaning, moving of data to a wastebasket · CPC title

• G06F3/061

  Improving I/O performance · CPC title