Distributed metadata in a high performance computing environment

Patent metadata

Field	Value
Publication number	US-9703788-B1
Application number	US-201414230800-A
Country	US
Kind code	B1
Filing date	Mar 31, 2014
Priority date	Mar 31, 2014
Publication date	Jul 11, 2017
Grant date	Jul 11, 2017

Abstract

A computer-executable method, system, and computer program product for managing meta-data in a distributed storage system, wherein the distributed storage system includes one or more burst buffers enabled to operate with a distributed key-value store, the co computer-executable method, system, and computer program product comprising receiving a request for meta-data associated with a block of data stored in a first burst buffer of the one or more burst buffers in the distributed storage system, wherein the meta data is associated with a key-value, determining which of the one or more burst buffers stores the requested metadata, and upon determination that a first burst buffer of the one or more burst buffers stores the requested metadata, locating the key-value in a portion of the distributed key-value store accessible from the first burst buffer.

First claim

What is claimed is:

1. A computer-executable method of managing meta-data for a high performance computing (HPC) environment in a distributed storage system, wherein the distributed storage system includes one or more burst buffers enabled to operate with a distributed key-value store and one or more data storage arrays, the computer-executable method comprising:
receiving a request for meta-data associated with a block of data stored in a first burst buffer of the one or more burst buffers in the distributed storage system, wherein the meta data is associated with a key-value;
determining which of the one or more burst buffers stores the requested metadata;
upon determination that a first burst buffer of the one or more burst buffers stores the requested metadata, locating the key-value in a portion of the distributed key-value store accessible from the first burst buffer;
communicating between each of the one or more burst buffers to manage the distributed key-value store;
moving data between the one or more burst buffers and the one or more data storage arrays, in the distributed storage system, based on a frequency of use of data;
wherein the determination enables querying for the metadata in a portion of a total amount of metadata on the distributed key-value store;
determining a state of the distributed key-value store; and
based on the state, re-balancing the distributed key-value store.

2. The computer-executable method of claim 1 , wherein the distributed key-value store is a Multidimensional Data Hashing Indexing Middleware (MDHIM).

3. The computer-executable method of claim 1 , wherein the determining comprises:
accessing which of the one or more burst buffers stores a range of key-values, wherein the range includes the key-value.

4. A system, comprising:
a distributed storage system, wherein the distributed storage system includes one or more burst buffers enabled to operate with a distributed key-value store and one or more data storage arrays; and
computer-executable program logic encoded in memory of one or more computers in communication with the distributed storage system to enable management of metadata for a high performance computing (HPC) environment in the distributed storage system, wherein the computer-executable program logic is configured for the execution of:
receiving a request for meta-data associated with a block of data stored in a first burst buffer of the one or more burst buffers in the distributed storage system, wherein the meta data is associated with a key-value;
determining which of the one or more burst buffers stores the requested metadata;

upon determination that a first burst buffer of the one or more burst buffers stores the requested metadata, locating the key-value in a portion of the distributed key-value store accessible from the first burst buffer;
communicating between each of the one or more burst buffers to manage the distributed key-value store;
moving data between the one or more burst buffers and the one or more data storage arrays, in the distributed storage system, based on a frequency of use of data;
wherein the determination enables querying for the metadata in a portion of a total amount of metadata on the distributed key-value store;
determining a state of the distributed key-value store; and
based on the state, re-balancing the distributed key-value store.

5. The system of claim 4 , wherein the distributed key-value store is a Multidimensional Data Hashing Indexing Middleware (MDHIM).

6. The system of claim 4 , wherein the determining comprises:
accessing which of the one or more burst buffers stores a range of key-values, wherein the range includes the key-value.

7. A computer program product for managing meta-data for a high performance computing (HPC) environment in a distributed storage system, wherein the distributed storage system includes one or more burst buffers enabled to operate with a distributed key-value store and one or more data storage arrays, the computer program product comprising:
a non-transitory computer readable medium encoded with computer-executable program code for using read signatures in replication, the code configured to enable the execution of:
receiving a request for meta-data associated with a block of data stored in a first burst buffer of the one or more burst buffers in the distributed storage system, wherein the meta data is associated with a key-value;
determining which of the one or more burst buffers stores the requested metadata;
upon determination that a first burst buffer of the one or more burst buffers stores the requested metadata, locating the key-value in a portion of the distributed key-value store accessible from the first burst buffer;
communicating between each of the one or more burst buffers to manage the distributed key-value store;
moving data between the one or more burst buffers and the one or more data storage arrays, in the distributed storage system, based on a frequency of use of data;
wherein the determination enables querying for the metadata in a portion of a total amount of metadata on the distributed key-value store;
determining a state of the distributed key-value store; and
based on the state, re-balancing the distributed key-value store.

8. The computer program product of claim 7 , wherein the distributed key-value store is a Multidimensional Data Hashing Indexing Middleware (MDHIM).

9. The computer program product of claim 7 , wherein the determining comprises:
accessing which of the one or more burst buffers stores a range of key-values, wherein the range includes the key-value.

Assignees

• Emc Corp
• Emc Ip Holding Co Llc

Inventors

• Bent John M
• Faibish Sorin
• Zhang Zhenhua
• Liu Xuezhao
• Tang Haiying

Classifications

• G06F17/3056

  Physics · mapped topic

• G06F17/30194

  Physics · mapped topic

• G06F17/30563

  Physics · mapped topic

• G06F17/30094 Primary

  Physics · mapped topic

• G06F17/30445

  Physics · mapped topic