Data storage, distribution, reconstruction and recovery methods and devices, and data processing system

Patent metadata

Field	Value
Publication number	US-11010072-B2
Application number	US-201816495042-A
Country	US
Kind code	B2
Filing date	Mar 16, 2018
Priority date	Mar 17, 2017
Publication date	May 18, 2021
Grant date	May 18, 2021

Abstract

Embodiments of the present application provide a method and apparatus for displaying objects. In the method, data segments of to-be-stored data are stored in storage objects. A storage object is a minimum unit for data storage. The difference between the number of data segments stored in a storage object and that of another storage object is no more than a first preset threshold. Data segments are equally stored, to the greatest extent, into storage objects. The case that data having segments lost due to failure of a storage object cannot be restored is avoided.

First claim

The invention claimed is:

1. A method for data storage, comprising:
assigning x storage objects to to-be-stored data according to a predefined erasure coding strategy of k+m; wherein, k is the number of original segments, m is the number of redundant segments, x is greater than 1 and not greater than k+m, and a storage object is a minimum unit for data storage;
performing data division and redundancy processing to the to-be-stored data according to the erasure coding strategy of k+m, to obtain k+m data segments;
storing the k+m data segments into the x storage objects, wherein, the difference between the number of data segments stored in a storage object and that of another storage object is no more than a first preset threshold;
recording index information for the to-be-stored data, the index information comprising correspondence between each of data segments and a storage object in which the data segment is stored;
for each piece of the recorded index information, determining whether invalid data segments associated with the piece of recorded index information are more than a fourth predefined threshold, if so, determining the piece of recorded index information as index information for recovery; wherein, the invalid data segments are data segments stored in fault storage objects;
determining valid data segments associated with the index information for recovery as to-be-recovered data segments, the valid data segments being data segments in the index information for recovery that correspond to fault-free storage objects;
assigning storage objects to the to-be-recovered data segments; and
storing the to-be-recovered data segments into the assigned storage objects.

2. The method according to claim 1 , wherein, assigning x storage objects to to-be-stored data according to a predefined erasure coding strategy of k+m comprises:
determining whether available storage nodes are fewer than k+m;
if not fewer than k+m, determining x storage objects from the available storage nodes; wherein, each of the determined storage objects is from a different storage node, and x is equal to k+m;
storing the k+m data segments into x storage objects comprises:
storing each of the data segments into a storage object in a different storage node.

3. The method according to claim 2 , wherein, if the available storage nodes are fewer than k+m, the method further comprises:
determining whether available storage objects of all the available storage nodes are fewer than k+m;
if not fewer than k+m, determining x storage objects from all the available storage nodes; wherein, x is equal to k+m, and the difference between the number of data segments stored in a storage node and that of another storage node is no more than a second preset threshold;
storing the k+m data segments into x storage objects comprises:
storing each of the data segments into a different storage object.

4. The method according to claim 3 , wherein, if the available storage objects of all the available storage nodes are fewer than k+m, the method further comprises:
assigning all the available storage objects to the to-be-stored data; wherein x is equal to the number of all the available storage objects;
storing the k+m data segments into x storage objects comprises:
equally dividing the k+m data segments into x parts, and storing respectively the x parts of data into the x storage objects.

5. The method of claim 1 , further comprising:
for each piece of the recorded index information, determining record counts of storage objects associated with the piece of index information, wherein, a record count of a storage object associated with a piece of index information is the number of data segments stored in the storage object that are associated with the piece of index information;
determining a piece of index information having a target storage object whose record count is greater than a third predefined threshold as index information for distribution;
determining, among the data segments stored in the target storage object, to-be-distributed data segments;
assigning distributed storage objects to the to-be-distributed data segments; and
storing the to-be-distributed data segments into the distributed storage objects.

6. The method of claim 1 , further comprising:
determining index information for construction in the recorded index information; wherein, the index information for reconstruction contains information of a fault storage object;
reading, according to information of fault-free storage objects contained in the index information for reconstruction, target data segments from the fault-free storage objects, the target data segments being data segments of the data to be reconstructed associated with the information index for reconstruction;
obtaining a restored segment through reconstruction with the target data segments; and
storing the restored segment into an assigned storage object.

7. An apparatus for data storage, comprising:
a first assigning module, configured for assigning x storage objects to to-be-stored data according to a predefined erasure coding strategy of k+m; wherein, k is the number of original segments, m is the number of redundant segments, x is greater than 1 and not greater than k+m, and a storage object is a minimum unit for data storage;
a division module, configured for performing data division and redundancy processing to the to-be-stored data according to the erasure coding strategy of k+m, to obtain k+m data segments;
a first storing module, configured for storing the k+m data segments into the x storage objects, wherein, the difference between the number of data segments stored in a storage object and that of another storage object is no more than a first preset threshold;
a first recording module, configured for recording index information for the to-be-stored data, the index information comprising correspondence between each of data segments and a storage object in which the data segment is stored;
a fourth determining module, configured for, for each piece of the recorded index information, determining whether invalid data segments associated with the piece of recorded index information are more than a fourth predefined threshold, if so, determining the piece of recorded index information as index information for recovery; wherein, the invalid data segments are data segments stored in fault storage objects;
a fifth determining module, configured for determining valid data segments associated with the index information for recovery as to-be-recovered data segments, the valid data segments being data segments in the index information for recovery that correspond to fault-free storage objects;
a third assigning module, configured for assigning storage objects to the to-be-recovered data segments; and
a fourth storing module, configured for storing the to-be-recovered data segments into the assigned storage objects.

8. The apparatus according to claim 7 , wherein, the first assigning module comprises:
a first decision sub-module, configured for determining whether available storage nodes are fewer than k+m; and
a first determining sub-module, configured for determining, if the available storage nodes are not fewer than k+m, x storage objects from the available storage nodes; wherein, each of the determined storage objects is from a different storage node, and x is equal to k+m; and
the first storing module comprises:
a first storing sub-module, configured for storing each of the data segments into a storage object in a different storage node if the available storage nodes are not fewer than k+m.

9. The apparatus according to claim 8 , wherein, the first assigning module comprises:
a second decision sub-module, c

Assignees

• Hangzhou Hikvision Digital Tec

Inventors

• Ye Min
• Lin Peng
• Wang Weichun
• Lin Qiqian

Classifications

• G06F3/067

  Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS] · CPC title

• G06F3/0619

  in relation to data integrity, e.g. data losses, bit errors · CPC title

• G06F3/0689

  Disk arrays, e.g. RAID, JBOD · CPC title

• G06F3/065

  Replication mechanisms · CPC title

• G06F3/062 Primary

  Securing storage systems · CPC title