Data storage method, apparatus, and system

Patent metadata

Field	Value
Publication number	US-11487618-B2
Application number	US-202117153810-A
Country	US
Kind code	B2
Filing date	Jan 20, 2021
Priority date	Dec 20, 2017
Publication date	Nov 1, 2022
Grant date	Nov 1, 2022

Abstract

A storage client needs to store to-be-written data into a distributed storage system, and storage nodes corresponding to a first data unit assigned for the to-be-written data by a management server are only some nodes in a storage node group. When receiving a status of the first data unit returned by the management server, the storage client may determine quantities of data blocks and parity blocks needing to be generated during EC coding on the to-be-written data. The storage client stores the generated data blocks and parity blocks into some storage nodes designated by the management server in a partition where the first data unit is located. Accordingly, dynamic adjustment of an EC redundancy ratio is implemented, and the management server may exclude some nodes in the partition from a storage range of the to-be-written data based on a requirement, thereby reducing a data storage IO amount.

First claim

What is claimed is:

1. A data storage system, wherein the data storage system comprises:
a storage client;
a management server; and
a storage node group includes N+M storage nodes for storing data according to a erasure coding (EC) technology, wherein
when the N+M storage nodes are in a normal state, the N storage nodes are configured to store data blocks;
the M storage nodes are configured to store parity blocks generated based on the data blocks with the EC technology; and
N and M are positive integers greater than 2;

the storage client is configured to:
obtain a quantity of the storage nodes that the storage client is allowed to use in the storage group from the management server, wherein the quantity of the storage nodes is less than N+M;
generate a plurality of data blocks by dividing data and one or more parity blocks for the plurality of data blocks, wherein a quantity of the plurality of data blocks and the one or more parity blocks are the same as the quantity of the storage nodes; and
store the plurality of data blocks and the one or more parity blocks to the storage nodes that the storage client is allowed to use in the storage group; and
store the quantity of storage nodes in a first data unit in a first partition of the storage node group.

2. The system according to claim 1 , wherein the management server is further configured to record a status of the N+M storage nodes in the storage node group.

3. The system according to claim 2 , wherein the storage client is further configured to obtain the status from the management server.

4. The system according to claim 1 , wherein the quantity of the one or more parity blocks is equal to M, and the quantity of the plurality of data blocks is equal to a value obtained by subtracting M from the quantity of the storage nodes the storage client is allowed to use.

5. A data storage method in a data storage system, wherein the data storage system comprises:
a storage client;
a management server; and
a storage node group includes N+M storage nodes for storing data according to an erasure coding (EC) technology, wherein
when the N+M storage nodes are in a normal state, the N storage nodes are configured to store data blocks;
the M storage nodes are configured to store parity blocks generated based on the data blocks with the EC technology; and
N and M are positive integers greater than 2; and, wherein the method comprising:

obtaining, by the storage client, a quantity of the storage nodes that the storage client is allowed to use in the storage group from the management server, wherein the quantity of the storage nodes is less than N+M;
generating, by the storage client, a plurality of data blocks by dividing data and one or more parity blocks for the plurality of data blocks;
a quantity of the plurality of data blocks and the one or more parity blocks are the same as the quantity of the storage nodes;
storing, by the storage client, the plurality of data blocks and the one or more parity blocks to the storage nodes that the storage client is allowed to use in the storage group; and
storing the quantity of storage nodes in a first data unit in a first partition of the storage node group.

6. The method according to claim 5 , wherein the method further comprises:
recording, by the management server, a status of the N+M storage nodes in the storage node group.

7. The method according to claim 6 , wherein the method further comprises:
obtaining, by the storage client, the status from the management server.

8. The method according to claim 5 , wherein the quantity of the one or more parity blocks is equal to M, and the quantity of the plurality of data blocks is equal to a value obtained by subtracting M from the quantity of the storage nodes the storage client is allowed to use.

9. A non-volatile and non-transitory computer readable storage medium, wherein the computer readable storage medium stores a computer instruction applied to a data storage system, wherein the data storage system comprises:
a storage client;
a management server; and
a storage node group includes N+M storage nodes for storing data according to an erasure coding (EC) technology, wherein
when the N+M storage nodes are in a normal state, the N storage nodes are configured to store data blocks;
the M storage nodes are configured to store parity blocks generated based on the data blocks with the EC technology; and
N and M are positive integers greater than 2; and, wherein

when the computer instruction executed by a computer as the storage client to cause the computer perform:
obtaining a quantity of the storage nodes that the storage client is allowed to use in the storage group from the management server, wherein the quantity of the storage nodes is less than N+M;
generating a plurality of data blocks by dividing data and one or more parity blocks for the plurality of data blocks, wherein a quantity of the plurality of data blocks and the one or more parity blocks is the same as the quantity of the storage nodes;
storing the plurality of data blocks and the one or more parity blocks to the storage nodes that the storage client is allowed to use in the storage group; and
storing the quantity of storage nodes in a first data unit in a first partition of the storage node group.

10. The non-volatile and non-transitory computer readable storage medium according to claim 9 , wherein the quantity of the one or more parity blocks is equal to M, and the quantity of the plurality of data blocks is equal to a value obtained by subtracting M from the quantity of the storage nodes the storage client is allowed to use.

Assignees

• Huawei Cloud Computing Tech Co Ltd

Inventors

• Liu Xiaowei
• Wu Huatao
• Yin Lihui

Classifications

• G06F11/1044

  with specific ECC/EDC distribution · CPC title

• G06F3/0647

  Migration mechanisms · CPC title

• G06F9/5061

  Partitioning or combining of resources · CPC title

• G06F3/0644

  Management of space entities, e.g. partitions, extents, pools · CPC title

• G06F11/1004 Primary

  to protect a block of data words, e.g. CRC or checksum (G06F11/1076 takes precedence; security arrangements for protecting computers or computer systems against unauthorized activity G06F21/00) · CPC title