Assignment of data within file systems

Patent metadata

Field	Value
Publication number	US-10127237-B2
Application number	US-201514974477-A
Country	US
Kind code	B2
Filing date	Dec 18, 2015
Priority date	Dec 18, 2015
Publication date	Nov 13, 2018
Grant date	Nov 13, 2018

Abstract

The embodiments relate to assigning data to processors of a file system. Metadata associated with respective blocks of data, and an initial batch of the blocks is assigned to nodes of a file system based on the metadata. Unassigned blocks are selectively assigned to one or more of the nodes. The selective assignment includes constructing a linear regression model based on node data, and determining a value for each node based on the linear regression model. Each value is associated with a predicted load corresponding to a new assignment of one or more unassigned blocks.

First claim

What is claimed is:

1. A method comprising:
gathering metadata associated with respective blocks of data;
assigning an initial batch of the blocks to nodes of a file system based on the gathered metadata;
selectively assigning unassigned blocks to one or more of the nodes, the selective assignment comprising:
constructing a linear regression model based on node data; and
for each node, determining a value based on the linear regression model, wherein each value is associated with a predicted load corresponding to a new assignment of one or more unassigned blocks; and

initializing a control factor associated with the selective assignment, wherein the control factor controls a maximum permitted block load per node, and wherein the selective assignment of the unassigned blocks further comprises performing a first assignment circuit, including:
comparing a first predicted value associated with a first node to the control factor;
in response to determining that the predicted value associated with the first node is less than the control factor, assigning the new assignment to the first node; and
in response to determining that the predicted value associated with the first node exceeds the control factor, comparing a second predicted value associated with a second node to the control factor to determine if the new assignment is assignable to the second node.

2. The method of claim 1 , further comprising adjusting the control factor in response to determining that the predicted value for each node exceeds the control factor, and performing a second assignment circuit to the nodes based on the adjusted control factor.

3. The method of claim 1 , wherein constructing the model comprises:
initializing the model based on a set of initial values associated with predicted costs;
collecting one or more read metric samples for each of the nodes; and
dynamically refining the model based on the collected samples.

4. The method of claim 1 , further comprising refining the model with a decay factor, wherein the decay factor governs relativity of weight between past samples and a new sample.

5. The method of claim 1 , further comprising controlling operation of the selective assignment based on a threshold associated with a time budget, including comparing a remaining time to the threshold, and selecting a mode of operation of the selective assignment based on the comparison.

6. A computer program product comprising a computer readable hardware storage device having program code embodied therewith, the program code executable by a processing unit to:
gather metadata associated with respective blocks of data;
assign an initial batch of the blocks to nodes of a file system based on the gathered metadata;
selectively assign unassigned blocks to one or more of the nodes, including initialize a control factor associated with the selective assignment, wherein the control factor controls a maximum permitted block load per node, the selective assignment comprising program code to:
construct a linear regression model based on node data; and
each node, determine a value based on the linear regression model, wherein each value is associated with a predicted load corresponding to a new assignment of one or more unassigned blocks; and

wherein the selective assignment of the unassigned blocks further comprises program code to perform a first assignment circuit, including program code to:
compare a first predicted value associated with a first node to the control factor;
in response to determining that the predicted value associated with the first node is less than the control factor, assign the new assignment to the first node; and
in response to determining that the predicted value associated with the first node exceeds the control factor, compare a second predicted value associated with a second node to the control factor to determine if the new assignment is assignable to the second node.

7. The computer program product of claim 6 , further comprising program code to adjust the control factor in response to determining that the predicted value for each node exceeds the control factor, and perform a second assignment circuit to the nodes based on the adjusted control factor.

8. The computer program product of claim 6 , wherein constructing the model comprises program code to:
initialize the model based on a set of initial values;
collect one or more read metric samples for each of the nodes; and
dynamically refine the model based on the collected samples.

9. The computer program product of claim 6 , further comprising program code to refine the model with a decay factor, wherein the decay factor governs a relativity of weight between past samples and a new sample.

10. The computer program product of claim 6 , further comprising program code to control operation of the selective assignment based on a threshold associated with a time budget, including program code to compare a remaining time to the threshold, and select a mode of operation of the selective assignment of the unassigned blocks based on the comparison.

11. A system comprising:
a processing unit in communication with memory;
a plurality of nodes, each node having local persistent storage and a local processor; and
a tool in communication with the nodes, the tool to:
gather metadata associated with respective blocks of data;
assign an initial batch of the blocks to nodes of a file system based on the gathered metadata; and
selectively assign unassigned blocks to one or more of the nodes, the selective assignment comprising the tool to:
initialize a control factor associated with the selective assignment, wherein the control factor controls a maximum permitted block load per node, and wherein the selective assignment of the unassigned blocks further comprises the tool to perform a first assignment circuit, including the tool to:
construct a linear regression model based on node data;
for each node, determine a value based on the linear regression model, wherein each value is associated with a predicted load corresponding to a new assignment of one or more unassigned blocks;
compare a first predicted value associated with a first node to the control factor;
in response to determining that the predicted value associated with the first node is less than the control factor, assign the new assignment to the first node; and
in response to determining that the predicted value associated with the first node exceeds the control factor, compare a second predicted value associated with a second node to the control factor to determine if the new assignment is assignable to the second node, adjust the control factor, and perform a second assignment circuit to the nodes based on the adjusted control factor.

12. The system of claim 11 , wherein the plurality of nodes are comprised in a distributed file system.

13. The system of claim 11 , wherein constructing the model comprises the tool to:
initialize the model based on a set of initial values;
collect one or more read metric samples for each of the nodes; and
dynamically refine the model based on the collected samples, including the tool to refine the model with a decay factor, wherein the decay factor governs a relativity of weight between past samples and a new sample.

14. The system of claim 11 , further comprising the tool to control operation of the selective assignment based on a threshold associated with a time budget, including the tool to compare a remaining time to the threshold, and select a mode of operation of the selective assignment of the unassigned blocks based on the comparison.

Assignees

• IBM

Inventors

• Jain Uttam
• Megiddo Nimrod
• Minhas Umar F
• Ozcan Fatma
• Van Der Linden Robbert

Classifications

• G06F17/30194

  Physics · mapped topic

• G06F17/30097 Primary

  Physics · mapped topic

• G06F16/182

  Distributed file systems · CPC title

• G06F16/137 Primary

  Hash-based (content-based indexing of textual data G06F16/31) · CPC title