Distributed shared log for modern storage servers

Patent metadata

Field	Value
Publication number	US-9807168-B2
Application number	US-201514684132-A
Country	US
Kind code	B2
Filing date	Apr 10, 2015
Priority date	Apr 10, 2015
Publication date	Oct 31, 2017
Grant date	Oct 31, 2017

Abstract

Disclosed herein are system, method, and computer program product embodiments for accessing a distributed shared log that supports a total order over log writes and multiple logical log streams. An embodiment operates by selecting a range of log entries for a logical log stream from the distributed shared log and mapping the selected range of log entries to associated storage units in a plurality of storage units. The selected range of log entries are stored at the associated storage units according to metadata that indexes the selected range of log entries by logical log stream. The selected range of log entries are then requested from the associated storage units based on the mapping. The selected range of log entries for the logical log stream are then returned from the associated storage units based on a scan of the metadata. The received log entries are then reassembled.

First claim

What is claimed is:

1. A computer implemented method for accessing a distributed shared log, comprising:
selecting, by at least one processor, a range of log entries for a logical log stream from the distributed shared log, the distributed shared log being distributed over a plurality of storage units, the distributed shared log associated with a plurality of logical log streams, and the distributed shared log providing ordered writes across the plurality of logical log streams;
mapping, by the at least one processor, the selected range of log entries to associated storage units in the plurality of storage units;
requesting, by the at least one processor, the selected range of log entries from the associated storage units based on the mapping, wherein the selected range of log entries are stored according to metadata at the associated storage units;
receiving, by the at least one processor, the selected range of log entries from the associated storage units based on the requesting; and
reassembling, by the at least one processor, the received selected range of log entries for the logical log stream.

2. The method of claim 1 , wherein the associated storage units provide the selected range of log entries in parallel.

3. The method of claim 1 , wherein consecutive segments of the selected range of log entries are stored at different storage units in the plurality of storage units.

4. The method of claim 1 , wherein the selected range of log entries are stored using a first storage tier and a second storage tier.

5. The method of claim 4 , wherein the second storage tier is a distributed file system.

6. The method of claim 4 , wherein a log entry from the selected range of log entries stored in the first storage tier contains a pointer to an associated log entry in the second storage tier.

7. The method of claim 1 , wherein the associated storage units retrieve the selected range of log entries by scanning the metadata based on the logical log stream of the selected range of log entries.

8. The method of claim 1 , wherein the distributed shared log supports an arbitrary number of logical log streams.

9. The method of claim 1 , wherein the selected range of log entries are persistently stored with the metadata at the associated storage units based on the mapping.

10. A system, comprising:
a memory; and
at least one processor coupled to the memory and configured to:
select a range of log entries for a logical log stream from a distributed shared log, the distributed shared log being distributed over a plurality of storage units, the distributed shared log associated with a plurality of logical log streams, and the distributed shared log providing ordered writes across the plurality of logical log streams;
map the selected range of log entries to associated storage units in the plurality of storage units;
request the selected range of log entries from the associated storage units based on the mapping, wherein the selected range of log entries arc stored according to metadata at the associated storage units;
receive the selected range of log entries from the associated storage units based on the requesting; and
reassemble the received selected range of log entries for the logical log stream.

11. The system of claim 10 , wherein the associated storage units provide the selected range of log entries in parallel.

12. The system of claim 10 , wherein consecutive segments of the selected range of log entries are stored at different storage units in the plurality of storage units.

13. The system of claim 10 , wherein the selected range of log entries are stored using a first storage tier and a second storage tier.

14. The system of claim 13 , wherein the second storage tier is a distributed file system.

15. The system of claim 13 , wherein a log entry from the selected range of log entries stored in the first storage tier contains a pointer to an associated log entry in the second storage tier.

16. The system of claim 10 , wherein the associated storage units retrieve the selected range of log entries by scanning the metadata based on the logical log stream of the selected range of log entries.

17. The system of claim 10 , wherein the selected range of log entries are stored on a persistent storage with the metadata at the associated storage units based on the mapping.

18. A tangible computer-readable device having instructions stored thereon that, when executed by at least one computing device, causes the at least one computing device to perform operations comprising:
selecting a range of log entries for a logical log stream from the distributed shared log, the distributed shared log being distributed over a plurality of storage units, the distributed shared log associated with a plurality of logical log streams, and the distributed shared log providing ordered writes across the plurality of logical log streams;
mapping the selected range of log entries to associated storage units in the plurality of storage units;
requesting the selected range of log entries from the associated storage units based on the mapping, wherein the selected range of log entries are stored according to metadata at the associated storage units;
receiving the selected range of log entries from the associated storage units based on the requesting; and
reassembling the received selected range of log entries for the logical log stream.

19. The computer-readable device of claim 18 , wherein the associated storage units provide the selected range of log entries in parallel.

20. The computer-readable device of claim 18 , wherein consecutive segments of the selected range of log entries are stored at different storage units in the plurality of storage units.

21. The computer-readable device of claim 18 , wherein the selected range of log entries are stored using a first storage tier and a second storage tier.

22. The computer-readable device of claim 18 , wherein the associated storage units retrieve the selected range of log entries by scanning the metadata based on the logical log stream of the selected range of log entries.

23. The computer-readable device of claim 18 , wherein the selected range of log entries are persistently stored with the metadata at the associated storage units based on the mapping.

Assignees

• Auch Nathan
• Bumbulis Peter
• Goel Anil
• Pound Jeffrey
• Sap Se

Inventors

• Auch Nathan
• Bumbulis Peter
• Goel Anil
• Pound Jeffrey

Classifications

• G06F15/167

  using a common memory, e.g. mailbox · CPC title

• H04L67/1097 Primary

  for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS] · CPC title