Storage cluster and method that efficiently store small objects with erasure codes

Patent metadata

Field	Value
Publication number	US-10120576-B2
Application number	US-201615065025-A
Country	US
Kind code	B2
Filing date	Mar 9, 2016
Priority date	Mar 9, 2016
Publication date	Nov 6, 2018
Grant date	Nov 6, 2018

Abstract

Small objects are efficiently stored with erasure codes by combining a small object with other small objects and/or large objects to form a single large object for chunking, and providing early notification of permanent storage to the sources of the objects to prevent small objects from becoming stale while waiting for additional objects to be combined.

First claim

What is claimed is:

1. A storage cluster comprising:
a number of permanent storage units; and
a controller coupled to the number of permanent storage units, the controller having an input buffer and a non-volatile temporary memory, the controller configured to:
receive a first object;
determine that the first object is smaller than an unused portion of the input buffer;
add the first object to the input buffer, thereby decreasing the unused portion of the input buffer;
responsive to a latency condition occurring:
copy object data including the first object from the input buffer to the non-volatile temporary memory, and
automatically notify a source of the first object that the first object has been permanently stored,
wherein the latency condition occurs when the first object waits in the input buffer for more than a predefined time;

receive a second object;
responsive to a determination that the second object is larger than the unused portion of the input buffer that includes the first object, permanently store the second object, and all objects in the input buffer including the first object, in the permanent storage units using erasure codes; and
automatically notify a source of the second object that the second object has been permanently stored.

2. The storage cluster of claim 1 , wherein copying the object data creates a temporary copy of the first object in the non-volatile temporary memory.

3. The storage cluster of claim 2 , wherein a source of the first object is notified of a permanent storage of the first object subsequent to a creation of the temporary copy of the first object in the non-volatile temporary memory and prior to permanently storing the first object in the permanent storage units.

4. The storage cluster of claim 1 , wherein the object data copied to the non-volatile temporary memory responsive to the latency condition occurring is of a predefined amount of data.

5. The storage cluster of claim 1 , wherein permanently storing the second object, and all objects in the input buffer including the first object, in the permanent storage units using erasure codes includes:
combining the second object, and all objects in the input buffer including the first object, into a single object;
chunking the single object into a number of chunks;
encoding the number of chunks to form a number of erasure-encoded chunks; and
transferring the number of erasure-encoded chunks to the permanent storage units, wherein the permanent storage units permanently store the number of erasure-encoded chunks.

6. The storage cluster of claim 1 , wherein the input buffer is a volatile memory.

7. The storage cluster of claim 1 , wherein the controller is further configured to:
receive a third object that is larger than the input buffer, wherein the input buffer is empty; and
permanently store the third object in the permanent storage units using erasure codes.

8. A method of operating a storage cluster, the method comprising:
receiving a first object;
determining that the first object is smaller than an unused portion of an input buffer;
adding the first object to the input buffer, thereby decreasing the unused portion of the input buffer;
responsive to a latency condition occurring:
copying object data including the first object from the input buffer to a non-volatile temporary memory, and
automatically notifying a source of the first object that the first object has been permanently stored,
wherein the latency condition occurs when the first object waits in the input buffer for more than a predefined time;

receiving a second object;
responsive to a determination that the second object is larger than the unused portion of the input buffer that includes the first object, permanently storing the second object, and all objects in the input buffer including the first object, in non-volatile permanent memories using erasure codes; and
automatically notifying a source of the second object that the second object has been permanently stored.

9. The method of claim 8 , wherein copying the object data creates a temporary copy of the first object in the non-volatile temporary memory.

10. The method of claim 8 , wherein the object data copied to the non-volatile temporary memory responsive to the latency condition occurring is of a predefined amount of data.

11. The method of claim 8 , wherein a source of the first object is notified of a permanent storage of the first object subsequent to a creation of a temporary copy of the first object in the non-volatile temporary memory and prior to permanently storing the first object in the non-volatile permanent memories.

12. The method of claim 8 , wherein permanently storing the second object, and all objects in the input buffer including the first object, in the non-volatile permanent memories using erasure codes includes:
combining the second object, and all objects in the input buffer including the first object, into a single object;
chunking the single object into a number of chunks;
encoding the number of chunks to form a number of erasure-encoded chunks; and
transferring the number of erasure-encoded chunks to a number of non-volatile permanent memories, wherein the non-volatile permanent memories permanently store the number of erasure-encoded chunks.

13. The method of claim 8 , wherein the input buffer is a volatile memory.

14. The method of claim 8 , further comprising:
receiving a third object that is larger than the input buffer, wherein the input buffer is empty; and
permanently storing the third object in the non-volatile permanent memories using erasure codes.

15. A method of operating a controller, the method comprising:
receiving a first object that is smaller than an unused portion of an input buffer;
adding the first object to the input buffer, thereby decreasing the unused portion of the input buffer;
responsive to a latency condition occurring:
copying object data including the first object from the input buffer to a non-volatile temporary memory, and
automatically notifying a source of the first object that the first object has been permanently stored,
wherein the latency condition occurs when the first object waits in the input buffer for more than a predefined time;

receiving a second object;
determining that the second object is larger than the unused portion of the input buffer;
chunking the second object and all objects in the input buffer including the first object, as a single combined object, thereby forming combined chunks
encoding the combined chunks to form combined erasure-encoded chunks; and
automatically notifying a source of the second object that the second object has been permanently stored.

16. The method of claim 15 , wherein copying the object data creates a temporary copy of the first object in the non-volatile temporary memory.

17. The method of claim 15 , wherein the object data copied to the non-volatile temporary memory responsive to the latency condition occurring is of a predefined amount of data.

18. The method of claim 15 , wherein a source of the first object is notified of a permanent storage of the first object subsequent to a creation of a temporary copy of the first object in the non-volatile temporary memory and prior to permanently storing the first object in non-volatile permanent memories.

19. The method of claim 15 , wherein the input buffer is volatile memory.

20. The method of claim 15 , further comprising:
receiving a third object that is larger than the input

Assignees

• HGST Netherlands BV
• Western Digital Tech Inc

Inventors

• Manzanares Adam
• Pamies-Juarez Lluis
• Guyot Cyril
• KEYSER Koen De
• Christiaens Mark
• Mateescu Robert

Classifications

• G06F3/0656

  Data buffering arrangements · CPC title

• G06F3/0661

  Format or protocol conversion arrangements · CPC title

• G06F3/0653

  Monitoring storage devices or systems · CPC title

• G06F3/0608 Primary

  Saving storage space on storage systems · CPC title

• G06F3/0643

  Management of files · CPC title