Log-structured storage systems

Patent metadata

Field	Value
Publication number	US-11294881-B2
Application number	US-201916712696-A
Country	US
Kind code	B2
Filing date	Dec 12, 2019
Priority date	Sep 12, 2019
Publication date	Apr 5, 2022
Grant date	Apr 5, 2022

Abstract

Disclosed herein are methods, systems, and apparatus, including computer programs encoded on computer storage devices, for data processing and storage. One of the methods includes receiving, by a storage system, a plurality of write requests for storing a respective plurality of blockchain data in the storage system. Each of the plurality of the blockchain data includes: a value of one or more of a block, a transaction, or a state of a blockchain network, and a key corresponding to the value. The storage system appends the plurality of the blockchain data to a data log file according to a chronological order of the plurality of the blockchain data, and refrains from sorting the plurality of the blockchain data in the data log file according to respective keys of values in the plurality of the blockchain data.

First claim

What is claimed is:

1. A computer-implemented method comprising:
receiving, by a storage system of a network node of a blockchain network, a write request for storing a plurality of blockchain data into a blockchain in the storage system, wherein the blockchain comprises a plurality of blocks, wherein the plurality of blockchain data comprise block data that are to be stored in the plurality of blocks of the blockchain and that are generated by one or more network nodes in the blockchain network, and wherein each of the plurality of the blockchain data comprises:
a value of at least one block in the blockchain of the blockchain network, and
a key corresponding to the value;

appending, by the storage system, the plurality of the blockchain data to one or more data log files in the storage system according to a chronological order of the plurality of the blockchain data, wherein the one or more data log files are configured to store the plurality of the blockchain data, wherein the one or more data log files are stored in a first-tier storage device in the storage system, and wherein the one or more data log files store the plurality of the blockchain data, the plurality of the blockchain data have respective physical storage locations in the one or more data log files, and the respective physical storage locations of the plurality of the blockchain data are arranged according to the chronological order of the plurality of the blockchain data;
after appending the plurality of the blockchain data to the one or more data log files and when the plurality of the blockchain data are stored in the first-tier storage device, refraining from rearranging, within the first-tier storage device, the respective physical storage locations of the plurality of the blockchain data in the one or more data log files according to respective keys of values in the plurality of the blockchain data, wherein the respective physical storage locations of the plurality of the blockchain data remain arranged within the first-tier storage device according to the chronological order of the plurality of the blockchain data, and wherein the respective physical storage locations of the plurality of the blockchain data in the one or more data log files are all located within the first-tier storage device;
generating, by the storage system, indexes indicating the respective physical storage locations of the plurality of the blockchain data in the one or more data log files, wherein one or more index log files are configured to store the indexes;
writing, by the storage system, the indexes into an index log file of the one or more index log files, wherein the index log file is stored in the first-tier storage device, wherein the index log file further comprises a list indicating at least one deletable data log file in the one or more data log files, wherein the at least one deletable data log file is to be deleted from the first-tier storage device after the at least one deletable data log file is migrated to a second-tier storage device in the storage system;
creating a second index log file after the at least one deletable data log file is migrated to the second-tier storage device, wherein the second index log file comprises an index indicating a physical storage location of the at least one deletable data log file in the second-tier storage device, and wherein the storage system is redirected to use the second index log file to access the at least one deletable data log file in the second-tier storage device; and
in response to determining a seal condition is met, sealing, by the storage system, the index log file and refraining from writing additional index data into the index log file, wherein the seal condition comprises a predetermined threshold of a maximum size of the index log file.

2. The computer-implemented method of claim 1 , wherein the key comprises a hash value corresponding to the value.

3. The computer-implemented method of claim 1 , wherein the one or more data log files are append-only files.

4. The computer-implemented method of claim 1 , wherein the first-tier storage device in the storage system is in a multi-tier storage subsystem of the storage system that comprises a plurality of tiers of storage devices, and the first-tier storage device has a highest access speed among the plurality of tiers of storage devices.

5. The computer-implemented method of claim 4 , further comprising:
determining respective migration scores of the plurality of the blockchain data according to the chronological order of the plurality of the blockchain data; and
migrating the plurality of the blockchain data stored in the first-tier storage device into the second-tier storage device according to the respective migration scores, wherein the second-tier storage device has a lower access speed than the first-tier storage device.

6. The computer-implemented method of claim 5 , wherein a migration score of older blockchain data is higher than a migration score of newer blockchain data.

7. The computer-implemented method of claim 1 , wherein appending the plurality of the blockchain data to the one or more data log files according to the chronological order of the plurality of the blockchain data comprises:
appending the plurality of the blockchain data to the one or more data log files according to times when the plurality of the blockchain data are received by the storage system, or
appending the plurality of the blockchain data to the one or more data log files according to times when the plurality of the blockchain data are generated by the blockchain network.

8. The computer-implemented method of claim 1 , wherein the plurality of the blockchain data comprise:
first blockchain data comprising a first block in the blockchain of the blockchain network;
second blockchain data comprising a second block in the blockchain of the blockchain network, the second block is subsequent to the first block in the blockchain; and
wherein:
the first blockchain data have a first physical storage location in the one or more data log files, the second blockchain data have a second physical storage location in the one or more data log files, and the second physical storage location is subsequent to the first physical storage location.

9. The computer-implemented method of claim 1 , further comprising performing data tiering or migration based on the respective physical storage locations of the plurality of the blockchain data.

10. The computer-implemented method of claim 9 , wherein performing data tiering or migration based on the respective physical storage locations of the plurality of the blockchain data comprises moving the plurality of the blockchain data from a first tier storage device of the storage system to a second tier storage device of the storage system based on the respective physical storage locations of the plurality of the blockchain data that are arranged according to the chronological order of the plurality of the blockchain data, wherein the first tier storage device has a performance characteristic that is superior to a corresponding performance characteristic of the second tier storage device.

11. The computer-implemented method of claim 1 , wherein the one or more data log files store metadata related to the plurality of blocks of the blockchain, and the metadata comprise one or more of transaction hash values, transaction sequence values, block hash values, block numbers, snapshot version numbers, a cyclic redundancy check (CRC) code, or encryption information.

12. The computer-implemented method of claim 1 , wherein the blockchain data further comprise history state data that describe world states o

Assignees

• Advanced New Technologies Co Ltd

Inventors

• Tian Shikun

Classifications

• H04L9/50

  using hash chains, e.g. blockchains or hash trees · CPC title

• G06F16/13

  File access structures, e.g. distributed indices (arrangements of input from, or output to, record carriers G06F3/06) · CPC title

• G06F16/182

  Distributed file systems · CPC title

• G06F16/1815 Primary

  Journaling file systems · CPC title

• G06F3/0638

  Organizing or formatting or addressing of data · CPC title