Distributed storage network with coordinated partial task execution and methods for use therewith

Patent metadata

Field	Value
Publication number	US-9848044-B2
Application number	US-201414287499-A
Country	US
Kind code	B2
Filing date	May 27, 2014
Priority date	Jul 31, 2013
Publication date	Dec 19, 2017
Grant date	Dec 19, 2017

Abstract

A method includes receiving a task for execution by a plurality of distributed storage and task execution units A priority level is determined for the task. A plurality of coordinated partial task requests are generated and sent to the plurality of distributed storage and task execution units, wherein the plurality coordinated partial task requests indicate a plurality of coordinated partial tasks and the priority level. A plurality of partial task results are received in response to performance of the plurality of coordinated partial tasks by the plurality of distributed storage and task execution units. A task result for the task is generated based on the plurality of partial task results.

First claim

What is claimed is:

1. A method for execution by one or more processing modules of one or more computing devices of a dispersed storage and task (DST) network, the method comprises:
receiving a task for execution by a plurality of distributed storage and task execution units;
receiving and processing data associated with the task to produce encoded data slice groupings, wherein the processing partitions the data associated with the task into a plurality of data partitions and, for each data partition, error encodes the data partition to produce encoded data slices and groups the encoded data slices into the encoded data slice groupings;
partitioning the task into partial tasks, where each of the partial tasks includes a corresponding one of the encoded data slice groupings;
determining a priority level for the task based on an estimated performance impact for the task;
generating a plurality of coordinated partial task requests to the plurality of distributed storage and task execution units, wherein the plurality coordinated partial task requests indicate a plurality of coordinated partial tasks, the priority level, and the corresponding one of the encoded data slice groupings, wherein at least one of the plurality of distributed storage and task execution units communicates coordination information with at least one other of the plurality of distributed storage and task execution units and wherein the coordination information relates to execution of another plurality of coordinated partial tasks;
receiving a plurality of partial task results in response to performance of the plurality of coordinated partial tasks by the plurality of distributed storage and task execution units; and
generating a task result for the task based on the plurality of partial task results.

2. The method of claim 1 wherein at least one of the plurality of distributed storage and task execution units communicates at least one of the plurality of partial task results with at least one other of the plurality of distributed storage and task execution units.

3. The method of claim 2 wherein the at least one other of the plurality of distributed storage and task execution units generates at least one other of the plurality of partial task results based on the at least one of the plurality of partial task results received from the at least one of the plurality of distributed storage and task execution units.

4. The method of claim 1 wherein the task includes one of: aborted transaction clean up from at least one prior aborted task, a data migration, expired data clean up corresponding to stored data that has expired, a snapshot clean up, an index health check, a segment health check, a scan for missing ones of encoded data slices, and a system maintenance task.

5. The method of claim 1 wherein at least one of the plurality of distributed storage and task execution units communicates processing resource availability information with at least one other of the plurality of distributed storage and task execution units; wherein the processing resource availability information includes at least one of: a current processing utilization, an estimated processing utilization, indication of at least one pending one of the plurality of coordinated partial tasks, a completion forecast for the at least one pending one of the plurality of coordinated partial tasks, and a commencement forecast for at least one non-pending one of the plurality of coordinated partial tasks.

6. The method of claim 1 wherein each the plurality of distributed storage and task execution units performs its generated partial task upon its corresponding one of the encoded data slice groupings to produce the partial task results.

7. A dispersed storage and task (DST) client device comprises:
a processing system including a processor and a memory that is configured to:
receive a task for execution by a plurality of distributed storage and task execution units;
determine a priority level for the task based on an estimated performance impact for the task;
receive and process data associated with the task to produce encoded data slice groupings, wherein the process partitions the data associated with the task into a plurality of data partitions and, for each data partition, error encodes the data partition to produce encoded data slices and groups the encoded data slices into the encoded data slice groupings;

partition the task into partial tasks, where each of the partial tasks includes a corresponding one of the encoded data slice groupings;
generate a plurality of coordinated partial task requests to the plurality of distributed storage and task execution units, wherein the plurality coordinated partial task requests indicate a plurality of coordinated partial tasks, the priority level, and the corresponding one of the encoded data slice groupings, wherein at least one other of the plurality of distributed storage and task execution units generates at least one other of the plurality of partial task results based on the at least one of the plurality of partial task results received from the at least one of the plurality of distributed storage and task execution units;
receive a plurality of partial task results in response to performance of the plurality of coordinated partial tasks by the plurality of distributed storage and task execution units; and
generate a task result for the task based on the plurality of partial task results.

8. The DST client device of claim 7 wherein the task includes one of: aborted transaction clean up from at least one prior aborted task, a data migration, expired data clean up corresponding to stored data that has expired, a snapshot clean up, an index health check, a segment health check, a scan for missing ones of encoded data slices, and a system maintenance task.

9. The DST client device of claim 7 wherein at least one of the plurality of distributed storage and task execution units communicates processing resource availability information with at least one other of the plurality of distributed storage and task execution units; wherein the processing resource availability information includes at least one of: a current processing utilization, an estimated processing utilization, indication of at least one pending one of the plurality of coordinated partial tasks, a completion forecast for the at least one pending one of the plurality of coordinated partial tasks, and a commencement forecast for at least one non-pending one of the plurality of coordinated partial tasks.

10. The DST client device of claim 7 wherein at least one of the plurality of distributed storage and task execution units communicates coordination information with at least one other of the plurality of distributed storage and task execution units; wherein the coordination information relates to execution of another plurality of coordinated partial tasks.

11. The dispersed storage and task (DST) client device of claim 7 wherein each the plurality of distributed storage and task execution units performs its generated partial task upon its corresponding one of the encoded data slice groupings to produce the partial task results.

12. A non-transitory computer readable storage medium comprises:
at least one memory section that stores operational instructions that, when executed by one or more processing modules of one or more computing devices of a dispersed storage network (DSN), causes the one or more computing devices to:
receive a task for execution by a plurality of distributed storage and task execution units;
determine a priority level for the task based on an estimated performance impact for the task;
receive and process data ass

Assignees

• Cleversafe Inc
• IBM

Inventors

• Leggette Wesley
• Baptist Andrew
• Dhuse Greg
• Resch Jason K
• Grube Gary W

Classifications

• G06F3/0683

  Plurality of storage devices · CPC title

• H04L67/1097 Primary

  for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS] · CPC title

• G06F9/5066

  Algorithms for mapping a plurality of inter-dependent sub-tasks onto a plurality of physical CPUs (mappping at compile time, see G06F8/451) · CPC title

• G06F3/067

  Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS] · CPC title

• G06F11/1076

  Parity data used in redundant arrays of independent storages, e.g. in RAID systems · CPC title