Object storage system with metadata operation priority processing

Patent metadata

Field	Value
Publication number	US-10877810-B2
Application number	US-201816147576-A
Country	US
Kind code	B2
Filing date	Sep 29, 2018
Priority date	Sep 29, 2018
Publication date	Dec 29, 2020
Grant date	Dec 29, 2020

Abstract

Example object storage systems and methods provide priority metadata processing. Metadata operations are received in response to change events for at least one data object. The metadata operations may include system operations configured to manage changes to data objects and user-method operations configured to execute user-defined methods using the data objects. System operations are executed with a first priority in response to system operations with the first priority being available for processing. User-method operations are executed with a second priority in response to no metadata operations with the first priority being available for processing.

First claim

What is claimed is:

1. A system, comprising:
at least one processor;
at least one memory coupled to the at least one processor; and
a metadata processor configured to:
receive metadata operations related to change events for at least one data object stored in a plurality of storage nodes configured to receive and store data objects, wherein the metadata operations include:
system operations configured to manage changes to the data objects stored in the plurality of storage nodes; and
user-method operations configured to execute user-defined methods using the data objects stored in the plurality of storage nodes;

selectively forward the system operations to an execution queue;
selectively forward the user-method operations to a data log;
execute, responsive to the system operations being available for processing in the execution queue, the system operations;
selectively add, responsive to no system operations being available for processing in the execution queue, the user-method operations to the execution queue; and
execute, responsive to the user-method operations being available for processing in the execution queue, the user-method operations.

2. The system of claim 1 , further comprising:
a system operations processor configured to process the system operations from the execution queue.

3. The system of claim 1 , further comprising:
a user-method operations processor configured to process, responsive to adding the user-method operations to the execution queue, the user-method operations from the data log.

4. The system of claim 1 , further comprising:
a metadata streamer stored in the at least one memory for execution by the at least one processor and configured to:
identify the change events related to the data objects stored in the plurality of storage nodes; and
forward the metadata operations to an input queue of the metadata processor; and

an operations categorizer stored in the at least one memory for execution by the at least one processor and configured to:
identify the system operations and the user-method operations from the metadata operations in the input queue;
selectively forward, for the metadata processor, the system operations to the execution queue; and
selectively forward, for the metadata processor, the user-method operations to the data log.

5. The system of claim 1 , further comprising:
a user input module stored in the at least one memory for execution by the at least one processor and configured to:
receive a user-defined method request; and
parse the user-defined method request into at least one user-method operation; and
a job manager stored in the at least one memory for execution by the at least one processor and configured to submit the at least one user-method operation from the user input module to the metadata processor.

6. The system of claim 1 , wherein:
the system operations include operations related to user application access to the data objects stored in the plurality of storage nodes from at least one production user application; and
the user-method operations include metadata queries configured to generate reports related to aggregate features of the data objects stored in the plurality of storage nodes.

7. The system of claim 1 , wherein:
the user-method operations include at least one of metadata queries, metadata filters, metadata formatting, or system-supported metadata actions;
a user selects the user-method operations for processing outside of operations related to user application access to data objects; and
the metadata processor is further configured to store results of the user-method operations in a system metadata repository.

8. The system of claim 1 , wherein the metadata processor is further configured to execute the metadata operations in the execution queue in a sequential first-in-first-out order.

9. The system of claim 8 , wherein the metadata processor is further configured to selectively add the user-method operations to the execution queue from the data log in response to the execution queue being empty.

10. A computer-implemented method, comprising:
receiving metadata operations in response to change events for at least one data object stored in a plurality of storage nodes, wherein the metadata operations include:
system operations configured to manage changes to data objects in the plurality of storage nodes; and
user-method operations configured to execute user-defined methods using the data objects in the plurality of storage nodes;

selectively forwarding the system operations to an execution queue;
selectively forwarding the user-method operations to a data log;
executing, responsive to the system operations being available for processing in the execution queue, the system operations;
selectively adding, responsive to no system operations being available for processing in the execution queue, the user-method operations to the execution queue; and
executing, responsive to the user-method operations being available for processing in the execution queue, the user-method operations.

11. The computer-implemented method of claim 10 , further comprising:
selecting new entries for the system operations from an input queue; and
sending, from the execution queue, the selected new entries to a system operations processor configured to process the system operations.

12. The computer-implemented method of claim 10 , further comprising:
selecting log entries from the data log in response to an input queue being empty; and
sending, from the execution queue, the selected log entries to a user-method operations processor configured to process the user-method operations from the data log.

13. The computer-implemented method of claim 10 , further comprising:
identifying change events related to the data objects stored in the plurality of storage nodes;
forwarding, responsive to the identified change events, the metadata operations to an input queue of a metadata processor; and
identifying the system operations and the user-method operations from the metadata operations in the input queue.

14. The computer-implemented method of claim 10 , wherein executing the metadata operations in the execution queue is in a sequential first-in-first-out order.

15. The computer-implemented method of claim 14 , wherein selectively adding the user-method operations to the execution queue from the data log is in response to the execution queue being empty.

16. The computer-implemented method of claim 10 , further comprising:
receiving a user-defined method request;
parsing the user-defined method request into at least one user-method operation; and
submitting the at least one user-method operation to a metadata processor.

17. The computer-implemented method of claim 10 , wherein:
the system operations include operations related to user application access to the data objects in the plurality of storage nodes from at least one production user application; and
the user-method operations include metadata queries configured to generate reports related to aggregate features of the data objects in the plurality of storage nodes.

18. The computer-implemented method of claim 10 ,
wherein the user-method operations include at least one of metadata queries, metadata filters, metadata formatting, or system-supported metadata actions; and
further comprising:
selecting by a user the user-method operations for processing outside of operations related to user application access to data objects; and
storing results of the user-method operations in a

Assignees

• Western Digital Tech Inc

Inventors

• Pyati Ameet
• Alam Muhammad Tanweer

Classifications

• G06F3/067

  Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS] · CPC title

• G06F16/22

  Indexing; Data structures therefor; Storage structures · CPC title

• G06F9/5038 Primary

  considering the execution order of a plurality of tasks, e.g. taking priority or time dependency constraints into consideration (scheduling strategies G06F9/4881 and subgroups) · CPC title

• G06F3/0659

  Command handling arrangements, e.g. command buffers, queues, command scheduling · CPC title

• H04L67/1097

  for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS] · CPC title