Storage of log-structured data

Patent metadata

Field	Value
Publication number	US-10671572-B2
Application number	US-201615182030-A
Country	US
Kind code	B2
Filing date	Jun 14, 2016
Priority date	Jun 14, 2016
Publication date	Jun 2, 2020
Grant date	Jun 2, 2020

Abstract

A system includes reception of a first write request from a client including a first key and a first one or more stream, payload pairs associated with the first key, copying of the first key and the first one or more stream, payload pairs to a first buffer of a volatile memory, storage of data of the first buffer in one or more blocks of a raw block non-volatile memory device, providing of the first buffer to a stream store server, reception of the first buffer at the stream store server, adding of the first key and the first one or more stream, payload pairs to a second buffer of the volatile memory, in key-order, storage of the data of the second buffer in a filesystem storage device, according to stream, and transmission of an indication of the durability of the key to the tail store server.

First claim

What is claimed is:

1. A system comprising:
a volatile memory;
a raw block non-volatile memory device;
a filesystem storage device; and
one or more processors to execute processor-executable process steps to cause the system to execute a tail store server and a stream store server,
the tail store server to:
receive a first write request from a client including a first log-sequence number and a first plurality of pairs associated with the first log-sequence number, each of the first plurality of pairs associated with a respective payload and stream identifier;
receive a second write request from the client including a second log-sequence number and a second plurality of pairs associated with the second log-sequence number, each of the second plurality of pairs associated with a respective payload and stream identifier;
copy the first log-sequence number the first plurality of pairs, the second log-sequence number and the second plurality of pairs to a first buffer of the volatile memory;
store data of the first buffer in one or more blocks of the raw block non-volatile memory device without using a filesystem; and
provide the first buffer to the stream store server,

the stream store server to:
receive the first buffer from the tail store server;
add the first log-sequence number, the first plurality of pairs, the second log-sequence number and the second plurality of pairs to a second buffer of the volatile memory;

determine that all log-sequence numbers of the second buffer span a contiguous numerical range;
in response to the determination that all log-sequence numbers of the second buffer a span a contiguous numerical range, generate a plurality of segments, where each segment comprises payloads of the second buffer associated with a single respective stream identifier and log-sequence numbers within the contiguous numerical range; and
store each segment in one or more respective filesystem files in the filesystem storage device.

2. A system according to claim 1 , the tail store server further to:
free the one or more blocks of the raw block non-volatile memory device in response to the indication of the durability transmitted by the stream store server.

3. A system according to claim 1 , wherein storage of the data of the first buffer in the one or more blocks of the raw block memory device comprises storage of the data in a file within one or more blocks of a partition of the raw block memory device,
wherein a header of each of the one or more blocks includes a pointer to a first page of a next one of the one or more blocks.

4. A system according to claim 3 , wherein the header of each of the one or more blocks includes the first log-sequence number.

5. A system according to claim 3 , wherein storage of the data of the first buffer in one or more blocks of the raw block non-volatile memory device comprises:
determination of a pointer to a first page of a last block of the one or more blocks;
storage, at the first page of the last block of the one or more blocks, of a new last block, the new last block comprising a header including a pointer to a first page of a next block; and
storage of the data within the new last block.

6. A system according to claim 1 , wherein storage of each segment in one or more respective filesystem files in the filesystem storage device comprises:
addition of a third log-sequence number and third plurality of pairs to the second buffer of the volatile memory;
determination that the second log-sequence number and the third log-sequence number are separated by a missing log-sequence number; and
in response to the determination that the second log-sequence number and the third log-sequence number are separated by a missing log-sequence number, addition of the missing log-sequence number to the second buffer, in numerical order, and storage of the data of the second buffer in the filesystem storage device.

7. A system according to claim 1 , wherein storage of the data of the local buffer in the filesystem storage device comprises:
storage of the data associated with a stream in a subdirectory associated with the first log-sequence number, the sub-directory stored within a directory associated with the stream identifier.

8. A method comprising:
receiving, by a tail store server, a first write request from a client including a first log-sequence number and a first plurality of pairs associated with the first log-sequence number, each of the first plurality of pairs associated with a respective payload and stream identifier;
receiving, by the tail store server, a second write request from the client including a second log-sequence number and a second plurality of pairs associated with the second log-sequence number, each of the second plurality of pairs associated with a respective payload and stream identifier;
copying the first log-sequence number, the first plurality of pairs, the second log-sequence number and the second plurality of pairs to a first buffer of a volatile memory;
storing data of the first buffer in one or more blocks of a raw block non-volatile memory device without using a filesystem;
providing the first buffer to a stream store server;
receiving the first buffer at the stream store server;
adding the first log-sequence number, the first plurality of pairs, the second log-sequence number and the second plurality of pairs to a second buffer of the volatile memory;
determining that all log-sequence numbers of the second buffer span a contiguous numerical range;
in response to determining that all log-sequence numbers of the second buffer a span a contiguous numerical range, generating a plurality of segments, where each segment comprises payloads of the second buffer associated with a single respective stream identifier and log-sequence numbers within the contiguous numerical range; and
storing each segment in one or more respective filesystem files in a filesystem storage device.

9. A method according to claim 8 , further comprising:
freeing the one or more blocks of the raw block non-volatile memory device in response to the indication of the durability transmitted by the stream store server.

10. A method according to claim 8 , wherein storing the data of the first buffer in the one or more blocks of the raw block memory device comprises storing the data in a file within one or more blocks of a partition of the raw block memory device,
wherein a header of each of the one or more blocks includes a pointer to a first page of a next one of the one or more blocks.

11. A method according to claim 10 , wherein the header of each of the one or more blocks includes the first log sequence number.

12. A method according to claim 10 , wherein storing the data of the first buffer in one or more blocks of the raw block non-volatile memory device comprises:
determining a pointer to a first page of a last block of the one or more blocks;
storing, at the first page of the last block of the one or more blocks, of a new last block, the new last block comprising a header including a pointer to a first page of a next block; and
storing of the data within the new last block.

13. A method according to claim 8 , wherein storing each segment in one or more respective filesystem files in the filesystem storage device comprises:
adding a third log-sequence number and third plurality of pairs to the second buffer of the volatile memory;
determining that the second log-sequence number and the third log-sequence number are separated by a missing log-sequence number; and
in response to the determining that second log-sequence number and the third log-sequence number are separate

Assignees

• Sap Se

Inventors

• Auch Nathan
• Maclean Scott
• Bumbulis Peter
• Pound Jeffrey
• Goel Anil Kumar

Classifications

• G06F16/2358

  Change logging, detection, and notification (replication G06F16/27) · CPC title

• G06F11/14

  Error detection or correction of the data by redundancy in operations (error detection or correction of the data by redundancy in hardware G06F11/16) · CPC title

• G06F16/1734

  Details of monitoring file system events, e.g. by the use of hooks, filter drivers, logs · CPC title

• G06F16/1805

  Append-only file systems, e.g. using logs or journals to store data · CPC title

• G06F16/1865 Primary

  Transactional file systems · CPC title