Object-based storage with garbage collection and data consolidation

Patent metadata

Field	Value
Publication number	US-12443559-B2
Application number	US-202519183475-A
Country	US
Kind code	B2
Filing date	Apr 18, 2025
Priority date	Nov 6, 2023
Publication date	Oct 14, 2025
Grant date	Oct 14, 2025

Abstract

Embodiments are directed to a file system that include object stores. An object store for write requests may be provided. Write ahead log (WAL) entries that include data blocks may be generated. A WAL object may be generated based on the WAL entries and stored in the object store. An in-memory overlay may be updated to associate the data blocks with the WAL object. A checkpoint operation may be executed to: generate an index object that includes index entries that associate other data blocks with data objects stored in the object store; update the index object to include index entries that associate the data blocks with the WAL object; store the updated index object in the object store; update the in-memory overlay to remove the association of the data blocks and the WAL object and update the in memory WAL to remove records of successfully checkpointed WAL objects.

First claim

What is claimed as new and desired to be protected by Letters Patent of the United States is:

1. A method for managing data in a file system over a network using one or more processors to execute instructions that are configured to cause performance of actions, comprising:
generating a write ahead log (WAL) object for an object store based on one or more WAL entries for one or more data blocks associated with the one or more write requests;
determining one or more types of index data structures to include the one or more data blocks with the one or more WAL entries, wherein the determination is based on the use of one or more of a rule, instruction or a library to select the one or more types from a plurality of different data structures; and
wherein an in-memory overlay is updated with one or more types of determined index data structures to associate the one or more data blocks with the WAL object;
determining a threshold for a number of stored WAL objects in the object store to initiate performance of further actions, including:
updating an index object to include one or more index entries that associate the one or more data blocks with the WAL object that is converted to a data object stored in the object store, wherein the conversion updates the in-memory overlay to remove the association of the one or more data blocks with the WAL object; and
using one or more additional financial costs charged by a cloud computing provider to initiate execution of garbage collection for one or more data objects stored in the object store.

2. The method of claim 1 , further comprising:
determining one or more of an amount of stored data objects being above a predetermined amount for the object store or a total amount of data objects in the datastore to identify the one or more additional costs.

3. The method of claim 1 , wherein the cloud computing provider provides access to an application programming interface (API) for the file system.

4. The method of claim 1 , wherein the index object further comprises: one or more other index entries that associate one or more other data blocks with one or more other data objects stored in the object store.

5. The method of claim 1 , wherein the conversion of the WAL object to the data object, further comprises:
referencing the WAL object in the updated index object; and
storing the updated index object in the object store.

6. The method of claim 1 , further comprising:
associating the one or more write requests with one or more write transactions, wherein each write request includes zero or more data blocks associated with a write transaction, and wherein the one or more write ahead log (WAL) entries include zero or more data blocks.

7. The method of claim 1 , further comprising:
initiating a checkpoint operation based on the determined threshold;
determining one or more globally committed WAL objects that correspond to one or more committed write transactions;
determining one or more locally committed WAL objects that are associated with one or more uncommitted write transactions; and
excluding the one or more locally committed WAL objects from the checkpoint operation.

8. A network computer, comprising:
a memory that stores at least instructions for managing data in a file system; and
one or more processors that execute instructions that are configured to cause performance of actions, including:
generating a write ahead log (WAL) object for an object store based on one or more WAL entries for one or more data blocks associated with the one or more write requests;
determining one or more types of index data structures to include the one or more data blocks with the one or more WAL entries, wherein the determination is based on the use of one or more of a rule, instruction or a library to select the one or more types from a plurality of different data structures; and
wherein an in-memory overlay is updated with one or more types of determined index data structures to associate the one or more data blocks with the WAL object;
determining a threshold for a number of stored WAL objects in the object store to initiate performance of further actions, including:
updating an index object to include one or more index entries that associate the one or more data blocks with the WAL object that is converted to a data object stored in the object store, wherein the conversion updates the in-memory overlay to remove the association of the one or more data blocks with the WAL object, and; and
using one or more additional financial costs charged by a cloud computing provider to initiate execution of garbage collection for one or more data objects stored in the object store.

9. The network computer of claim 8 , further comprising:
determining one or more of an amount of stored data objects being above a predetermined amount for the object store or a total amount of data objects in the datastore to identify the one or more additional costs.

10. The network computer of claim 8 , wherein the cloud computing provider provides access to an application programming interface (API) for the file system.

11. The network computer of claim 8 , wherein the index object further comprises:
one or more other index entries that associate one or more other data blocks with one or more other data objects stored in the object store.

12. The network computer of claim 8 , wherein the conversion of the WAL object to the data object, further comprises:
referencing the WAL object in the updated index object; and
storing the updated index object in the object store.

13. The network computer of claim 8 , further comprising:
associating the one or more write requests with one or more write transactions, wherein each write request includes zero or more data blocks associated with a write transaction, and wherein the one or more write ahead log (WAL) entries include zero or more data blocks.

14. The network computer of claim 8 , further comprising:
initiating a checkpoint operation based on the determined threshold;
determining one or more globally committed WAL objects that correspond to one or more committed write transactions;
determining one or more locally committed WAL objects that are associated with one or more uncommitted write transactions; and
excluding the one or more locally committed WAL objects from the checkpoint operation.

15. A processor readable non-transitory storage media that includes instructions for managing data in a file system over a network, wherein execution of the instructions by one or more processors on one or more network computers performs actions, comprising:
generating a write ahead log (WAL) object for an object store based on one or more WAL entries for one or more data blocks associated with the one or more write requests;
determining one or more types of index data structures to include the one or more data blocks with the one or more WAL entries, wherein the determination is based on the use of one or more of a rule, instruction or a library to select the one or more types from a plurality of different data structures; and
wherein an in-memory overlay is updated with one or more types of determined index data structures to associate the one or more data blocks with the WAL object;
determining a threshold for a number of stored WAL objects in the object store to initiate performance of further actions, including:
updating an index object to include one or more index entries that associate the one or more data blocks with the WAL object that is converted to a data object stored in the object store, wherein the conversion updates the in-memory

Assignees

• Qumulo Inc

Inventors

• Ellis Graham Edwin
• Fairweather Ying
• Garvin Thorne Davis
• Haber Steven Henry
• Bai Yuxi
• Chmiel Michael Anthony
• Kosakanchit Pathirat
• Maclaren Jonathan Michael
• Mcmullan Matthew Christopher
• Moody Tyler Morrison
• Passey Aaron James
• Phipps Rowan Arthur
• Rothschilds Thomas Gregory

Classifications

• G06F12/0253

  Garbage collection, i.e. reclamation of unreferenced memory · CPC title

• G06F2212/702

  Conservative garbage collection · CPC title

• G06F16/122 Primary

  using management policies (point-in-time backing up or restoration of persistent data G06F11/1446; file migration policies for HSM systems G06F16/185) · CPC title