Composable network-storage-based file systems

Patent metadata

Field	Value
Publication number	US-11907173-B1
Application number	US-202117457853-A
Country	US
Kind code	B1
Filing date	Dec 6, 2021
Priority date	Dec 6, 2021
Publication date	Feb 20, 2024
Grant date	Feb 20, 2024

Abstract

Systems and methods are provided for exposing disparate storage systems and technologies to compute instances through a unified file system interface that is agnostic to the underlying storage systems and technologies. A file system namespace service may provide the ability to compose a file system from different storage technologies and systems (e.g., block storage, object storage, etc.). The file system namespace service may also provide the ability to layer different storage services (e.g., by treating local block storage volume as a write back cache, by treating a file system as a read-through cache in front of an object storage service, etc.), and/or the ability to bind virtual files to other compute instances and/or storage systems.

First claim

What is claimed is:

1. A system comprising:
a host computing device comprising one or more processors and computer-readable memory, wherein the host computing device is configured to execute a compute instance and a namespace service client, and wherein the namespace service client receives file system commands generated by the compute instance using a file system application programming interface (API); and
a namespace service comprising one or more computing devices, wherein the namespace service is configured to:
receive, from the namespace service client, a namespace service request comprising data representing a file and directory in a namespace associated with the compute instance, wherein the namespace service request is based on a first file system command generated by the compute instance using the file system API;
determine, based at least partly on mapping data associated with the namespace, a network-based storage system mapped to the file, wherein the network-based storage system is to be accessed using a storage system API that is different from the file system API; and
send, to the namespace service client, a namespace service response comprising data representing the network-based storage system mapped to the file; and

wherein the host computing device is further configured to:
access data stored on the network-based storage system based at least partly on execution of a storage system operation on the network-based storage system mapped to the file, wherein the storage system operation is different from the first file system command; and
provide, to the compute instance, access to the data stored on the network-based storage system.

2. The system of claim 1 , wherein the host computing device is further configured to:
store resolution data regarding the namespace service response in a namespace service client cache;
determine, subsequent to storing the resolution data, that the compute instance is to access the file;
access the resolution data in the namespace service client cache; and
access data stored on the network-based storage system based at least partly on the resolution data.

3. The system of claim 1 ,
wherein the namespace service is further configured to:
receive, from the namespace service client, a second namespace service request comprising data representing a second file in the namespace;
determine, based at least partly on the mapping data, executable code mapped to the second file; and
send, to the namespace service client, a second namespace service response comprising data representing the executable code mapped to the second file; and

wherein the host computing device is further configured to:
send a request to execute the executable code;
receive output of the executable code; and
provide to the compute instance access to the output of the executable code as data associated with the second file.

4. The system of claim 1 ,
wherein the namespace service is further configured to:
receive, from the namespace service client, a second namespace service request comprising data representing a second file in the namespace;
determine, based at least partly on the mapping data, a second network-based storage system mapped to the second file, wherein the second network-based storage system is to be accessed using a second storage system API that is different from both the file system API and the storage system API;
send, to the namespace service client, a second namespace service response comprising data representing the second network-based storage system; and

wherein the host computing device is further configured to:
access second data stored on the second network-based storage system based at least partly on the second namespace service response and the second storage system API; and
provide, to the compute instance, access to the second data stored on the second network-based storage system using the file system API.

5. A computer-implemented method comprising:
under control of a computing system comprising memory and one or more computer processors configured to execute specific instructions:
receiving a first request representing a first file system access operation associated with a directory of a file system namespace;
determining, based at least partly on mapping data associated with the file system namespace, a network-based storage system mapped to the directory;
generating a first response to the first request based at least partly on execution of a storage system operation on the network-based storage system mapped to the directory, wherein the storage system operation is different from the first file system access operation;
receiving a second request representing a second file system access operation associated with a file of the file system namespace;
determining, based at least partly on the mapping data, executable code mapped to the file; and
generating a second response to the second request based at least partly on execution of the executable code mapped to the file, wherein execution of the executable code is different from execution of the second file system access operation.

6. The computer-implemented method of claim 5 , further comprising:
receiving an instantiation request to instantiate a compute instance, wherein the instantiation request is associated with namespace metadata comprising data regarding a file system of the compute instance; and
instantiating the compute instance on a host computing device, wherein the compute instance is associated with the file system namespace.

7. The computer-implemented method of claim 6 , further comprising:
generating the file system namespace;
analyzing the namespace metadata; and
based on analyzing the namespace metadata:
mapping the directory in the file system namespace to the network-based storage system; and
mapping the file in the file system namespace to the executable code.

8. The computer-implemented method of claim 7 , further comprising mapping write operations associated with the directory to a second storage system different from the network-based storage system, wherein mapping the directory to the network-based storage system comprises mapping read operations associated with the directory to the network-based storage system.

9. The computer-implemented method of claim 7 , further comprising mapping write operations associated with the directory to a local storage system of the host computing device, wherein data written based on a write operation to the local storage system is not written to the network-based storage system.

10. The computer-implemented method of claim 6 , further comprising:
executing, by the host computing device, an application within the compute instance;
generating, by the application, the first file system access operation; and
sending, by a namespace client subsystem of the host computing device, the first request to the computing system.

11. The computer-implemented method of claim 10 , further comprising:
receiving, by the namespace client subsystem, the first response from the computing system; and
caching, by the namespace client subsystem, the first response.

12. The computer-implemented method of claim 10 , further comprising:
sending, by the namespace client subsystem, a third request to execute the executable code;
receiving, by the namespace client subsystem, output of the executable code; and
providing, by the namespace client subsystem to the compute instance, access to the output of the executable code based at least partly on the second file system access operation, wherein access to output of the executable code is provided as data associated with the

Assignees

• Amazon Tech Inc

Inventors

• Warfield Andrew Kent
• Wires Jacob

Classifications

• G06F16/183 Primary

  Provision of network file services by network file servers, e.g. by using NFS, CIFS (network file access protocols H04L67/1097) · CPC title

• G06F9/547 Primary

  Remote procedure calls [RPC]; Web services · CPC title

• G06F16/172

  Caching, prefetching or hoarding of files · CPC title