Computing erasure metadata and data layout prior to storage using a processing platform

Patent metadata

Field	Value
Publication number	US-10135924-B2
Application number	US-201514751372-A
Country	US
Kind code	B2
Filing date	Jun 26, 2015
Priority date	Jun 26, 2015
Publication date	Nov 20, 2018
Grant date	Nov 20, 2018

Abstract

Techniques are provided for computing data and metadata layout prior to storage in a storage system using a processing platform. An exemplary processing platform comprises one or more of a compute node and a burst buffer appliance. The processing platform communicates with a plurality of the compute nodes over a network, wherein a plurality of applications executing on the plurality of compute nodes generate a plurality of data objects; computes erasure metadata for one or more of the data objects on at least one of the compute nodes; and provides the erasure metadata with the corresponding one or more data objects to a storage system. The processing platform optionally determines a full set of the data objects to be stored and queries the storage system to determine an anticipated layout of the full set of the data objects to be stored. The anticipated layout allows special handling, for example, for small files and large files that are identified based on predefined criteria.

First claim

What is claimed is:

1. An apparatus comprising:
a processing platform comprising a compute node;
the processing platform being configured to communicate with a plurality of said compute nodes over a network, wherein a plurality of applications executing on said plurality of compute nodes generate a plurality of data objects;
the processing platform further configured to compute erasure metadata for one or more of said data objects on at least one of said compute nodes and to provide said erasure metadata with said corresponding one or more data objects to a storage system.

2. The apparatus of claim 1 wherein said processing platform is further configured to determine a full set of said data objects to be stored.

3. The apparatus of claim 2 wherein said processing platform is further configured to query said storage system to determine an anticipated layout of said full set of said data objects to be stored.

4. The apparatus of claim 3 wherein said processing platform is further configured to shuffle at least a portion of one or more of said data objects generated by at least one of said plurality of said compute nodes to a second one of said plurality of said compute nodes based on said anticipated layout.

5. The apparatus of claim 3 wherein said data objects comprise small files and wherein said processing platform is further configured to aggregate a plurality of said data objects based on said anticipated layout, wherein said small files are identified based on a predefined size threshold.

6. The apparatus of claim 5 wherein said processing platform is further configured to create metadata for said aggregated plurality of said data objects and to provide said aggregated plurality of said data objects and said corresponding metadata to said storage system in a single batch operation.

7. The apparatus of claim 3 wherein at least one of said data objects comprise a large file and wherein said processing platform is further configured to write a portion of the large file to the storage system based on the anticipated layout and indicate to the storage system that the portion is a part of said large file, wherein said large file is identified based on a predefined size threshold.

8. The apparatus of claim 1 wherein said processing platform is further configured to write one or more of said data objects directly to said storage system using a remote direct memory access (RDMA) operation.

9. A method comprising:
configuring a processing platform to communicate with a plurality of compute nodes over a network, wherein a plurality of applications executing on said plurality of compute nodes generate a plurality of data objects, wherein said processing platform comprises one of said compute nodes; and
configuring the processing platform to compute erasure metadata for one or more of said data objects on at least one of said compute nodes and to provide said erasure metadata with said corresponding one or more data objects to a storage system.

10. The method of claim 9 further comprising the step of configuring said processing platform to determine a full set of said data objects to be stored.

11. The method of claim 10 further comprising the step of configuring said processing platform to query said storage system to determine an anticipated layout of said full set of said data objects to be stored.

12. The method of claim 11 further comprising the step of configuring said processing platform to shuffle at least a portion of one or more of said data objects generated by at least one of said plurality of said compute nodes to one or more of said data objects generated by a second one of said plurality of said compute nodes based on said anticipated layout.

13. The method of claim 11 wherein said data objects comprise small files and wherein said processing platform is further configured to aggregate a plurality of said data objects based on said anticipated layout, wherein said small files are identified based on a predefined size threshold.

14. The method of claim 13 further comprising the step of configuring said processing platform to create metadata for said aggregated plurality of said data objects and to provide said aggregated plurality of said data objects and said corresponding metadata to said storage system in a single batch operation.

15. The method of claim 11 wherein at least one of said data objects comprise a large file and wherein said processing platform is further configured to write a portion of the large file to the storage system based on the anticipated layout and indicate to the storage system that the portion is a part of said large file, wherein said large file is identified based on a predefined size threshold.

16. The method of claim 9 further comprising the step of configuring said processing platform to write one or more of said data objects directly to said storage system using a remote direct memory access (RDMA) operation.

17. A computer program product comprising a non-transitory processor-readable storage medium having encoded therein executable code of one or more software programs, wherein the one or more software programs when executed perform the steps of the method of claim 9 .

18. A compute architecture comprising:
a plurality of compute nodes;
a storage system; and
a processing platform comprising one of said compute nodes;
the processing platform being configured to communicate with a plurality of said compute nodes over a network, wherein a plurality of applications executing on said plurality of compute nodes generate a plurality of data objects;
the processing platform further configured to compute erasure metadata for one or more of said data objects on at least one of said compute nodes and to provide said erasure metadata with said corresponding one or more data objects to the storage system.

19. The compute architecture of claim 18 wherein said processing platform is further configured to query said storage system to determine an anticipated layout of said full set of said data objects to be stored.

20. The compute architecture of claim 18 wherein said processing platform is further configured to write one or more of said data objects directly to said storage system using a remote direct memory access (RDMA) operation.

Assignees

• Emc Ip Holding Co Llc
• Los Alamos Nat Security Llc

Inventors

• Bent John M
• Faibish Sorin
• Ting Dennis P J
• Teymouri Sasan
• Pedone Jr James M
• Grider Gary

Classifications

• G06F3/0689

  Disk arrays, e.g. RAID, JBOD · CPC title

• G06F3/0611

  in relation to response time · CPC title

• G06F3/067

  Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS] · CPC title

• H04L67/1097 Primary

  for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS] · CPC title

• G06F11/1076 Primary

  Parity data used in redundant arrays of independent storages, e.g. in RAID systems · CPC title