Data storage method, apparatus, and system

Patent metadata

Field	Value
Publication number	US-10915397-B2
Application number	US-201916382200-A
Country	US
Kind code	B2
Filing date	Apr 12, 2019
Priority date	Dec 20, 2017
Publication date	Feb 9, 2021
Grant date	Feb 9, 2021

Abstract

A storage client needs to store to-be-written data into a distributed storage system, and storage nodes corresponding to a first data unit assigned for the to-be-written data by a management server are only some nodes in a storage node group. When receiving a status of the first data unit returned by the management server, the storage client may determine quantities of data blocks and parity blocks needing to be generated during EC coding on the to-be-written data. The storage client stores the generated data blocks and parity blocks into some storage nodes designated by the management server in a partition where the first data unit is located. Accordingly, dynamic adjustment of an EC redundancy ratio is implemented, and the management server may exclude some nodes in the partition from a storage range of the to-be-written data based on a requirement, thereby reducing a data storage IO amount.

First claim

What is claimed is:

1. A data storage system, wherein the data storage system comprises:
a storage client; and
a distributed storage system, wherein the distributed storage system uses an erasure coding (EC) technology to store data and, the distributed storage system comprises:
a management server; and
a plurality of storage nodes, wherein:
the plurality of storage nodes comprise N storage nodes and M storage nodes;
the N storage nodes and the M storage nodes comprise a storage node group corresponding to one partition;
the N storage nodes are configured to store data blocks;
the M storage nodes are configured to store parity blocks generated based on the data blocks; and
N and M are positive integers greater than 2; and

wherein:
the storage client is configured to request the management server to assign a data unit for to-be-written data;
the management server is configured to:
assign a first data unit for the to-be-written data in a first partition, and
return an identifier of the first data unit and a status of the first data unit to the storage client, wherein:
the status of the first data unit indicates storage nodes the storage client is allowed to use in the first partition; and
a quantity of the storage nodes that the storage client is allowed to use is less than N+M,
wherein the storage nodes that the storage client is allowed to use in the first partition comprise non-faulty nodes; and

the storage client is further configured to:
obtain the identifier of the first data unit, the status of the first data unit, and a first partition view of the first partition, wherein the first partition view indicates a list of storage nodes comprised in the first partition;
determine, based on the status of the first data unit, quantities of data blocks and parity blocks that need to be generated, and perform EC coding on the to-be-written data, to generate data blocks and parity blocks corresponding to the to-be-written data, wherein a sum of quantities of the generated data blocks and parity blocks is equal to the quantity of the storage nodes the storage client is allowed to use; and
store, based on the first partition view and the status of the first data unit, the generated data blocks and parity blocks into the storage nodes the storage client is allowed to use in the first partition, wherein each node stores one data block or parity block.

2. The system according to claim 1 , wherein
the management server is further configured to record the status of the first data unit.

3. The system according to claim 1 , wherein:
the storage client is further configured:
to perform a read operation to obtain a status of a second data unit corresponding to to-be-read data, and a second partition view and a second partition status of a second partition in which the second data unit is located, wherein the status of the second data unit indicates storage nodes the storage client is allowed to use in the second partition when the management server creates the second data unit, and the second partition status indicates whether storage nodes comprised in the second partition are faulty currently; and

the storage client is further configured to:
determine, based on the status of the second data unit, the second partition view, and the second partition status, a list of normal storage nodes storing EC blocks corresponding to the to-be-read data; and
read, from the determined list of storage nodes, the EC blocks corresponding to the to-be-read data.

4. The system according to claim 1 , wherein before the storage client requests the management server to assign the data unit for the to-be-written data, the storage client is further configured to:
write, based on an identifier of a third data unit returned by the management server, EC blocks generated based on the to-be-written data to storage nodes comprised in a third partition in which the third data unit is located; and
when a write failure occurs, request the management server to reassign a data unit.

5. The system according to claim 1 , wherein
there is a faulty node in the first partition, the storage nodes that the storage client is allowed to use are non-faulty nodes in the first partition, and the status of the first data unit indicates states of storage nodes comprised in the first partition when the management server creates the first data unit; or
the storage nodes that the storage client is allowed to use are storage nodes that are designated in the first partition by the management server for the storage client and that store the to-be-written data.

6. The system according to claim 1 , wherein
a quantity of the generated parity blocks is equal to M, and a quantity of the generated data blocks is equal to a value obtained by subtracting M from the quantity of the storage nodes the storage client is allowed to use.

7. The system according to claim 1 , wherein the management server is further configured to:
determine, based on a quantity of partitions having a faulty node, whether partition degradation is allowed; and
assign the first data unit for the to-be-written data in the first partition when partition degradation is allowed.

8. The system according to claim 7 , wherein the management server is configured to:
determine whether a ratio of the quantity of the partitions having a faulty node to a total quantity of partitions is greater than a preset percentage; and
determine that partition degradation is allowed when the ratio is greater than the preset percentage.

9. The system according to claim 7 , wherein the management server is further configured to:
assign a fourth data unit for the to-be-written data in a fourth partition having no faulty node when partition degradation is not allowed; and
return an identifier of the fourth data unit to the storage client.

10. A storage client, wherein the storage client uses an erasure coding (EC) technology to store data into a distributed storage system, wherein the distributed storage system comprises:
a management server; and
a plurality of storage nodes, wherein:
the plurality of storage nodes comprises N storage nodes and M storage nodes;
the N storage nodes and the M storage nodes comprise a storage node group corresponding to one partition;
the N storage nodes are configured to store data blocks;
the M storage nodes are configured to store parity blocks generated based on the data blocks; and
N and M are positive integers greater than 2; and

wherein the storage client comprises an interface and a processor, and the interface communicates with the processor wherein the processor is configured to:
request the management server to assign a data unit for to-be-written data;
obtain an identifier of a first data unit, a status of the first data unit, and a first partition view of a first partition that are returned by the management server, wherein:
the first data unit is located in the first partition,
the first partition view indicates a list of storage nodes comprised in the first partition,
the status of the first data unit indicates storage nodes the storage client is allowed to use in the first partition, and
a quantity of the storage nodes that the storage client is allowed to use is less than N+M,
wherein the storage nodes that the storage client is allowed to use in the first partition comprise non-faulty nodes; and

determine, based on the status of the first data unit, quantities of data blocks and parity blocks that need to be generated, and perform EC coding on the to-be-written data, to generate data blocks and parity blocks corresponding to the to-be-written data, wherein a sum of quantities of the generated data blocks and pari

Assignees

• Huawei Tech Co Ltd

Inventors

• Liu Xiaowei
• Wu Huatao
• Yin Lihui

Classifications

• G06F11/1076 Primary

  Parity data used in redundant arrays of independent storages, e.g. in RAID systems · CPC title

• G06F9/5061

  Partitioning or combining of resources · CPC title

• G06F3/0652

  Erasing, e.g. deleting, data cleaning, moving of data to a wastebasket · CPC title

• G06F3/0619 Primary

  in relation to data integrity, e.g. data losses, bit errors · CPC title

• G06F11/1044

  with specific ECC/EDC distribution · CPC title