Distributed storage method and device

Patent metadata

Field	Value
Publication number	US-11301154-B2
Application number	US-201715851212-A
Country	US
Kind code	B2
Filing date	Dec 21, 2017
Priority date	Feb 6, 2016
Publication date	Apr 12, 2022
Grant date	Apr 12, 2022

Abstract

Embodiments of the present disclosure provide a distributed storage method and a distributed storage device. The method includes offline merging, by a first thread service of a distributed storage system, M small files in a file system, generating, by the first thread service, M pieces of metadata in the offline merging process, loading, by a second thread service of the distributed storage system, the M pieces of metadata into a metadata set, searching, by the second thread service, the metadata set for metadata of a first small file when the second thread service receives a first instruction, and performing, by the second thread service, the operation corresponding to the first instruction when the second thread service finds the metadata of the first small file in the metadata set.

First claim

What is claimed is:

1. A distributed storage method, comprising:
allocating different functions to different thread services of a distributed storage system, wherein a first thread service of the different thread services is responsible for offline merging M small files and generating metadata describing the M small files, wherein a second thread service of the different thread services is responsible for loading the metadata into a metadata set, searching within the metadata set, and performing operations, and wherein the first thread service and the second thread service are two different and separate thread services executed by the distributed storage system;
offline merging, by the first thread service of a distributed storage system, the M small files in a file system of the distributed storage system into one first file according to a first rule, wherein a size of each of the M small files is less than a predefined byte size, and wherein M is an integer that is greater than 1;
generating, by the first thread service, M pieces of metadata, wherein each of the M pieces of metadata indicates a location of one of the M small files in the first file;
loading, by the second thread service of the distributed storage system, the M pieces of metadata into the metadata set, wherein the metadata set comprises metadata describing a plurality of small files including the M small files that have been merged;
searching, by the second thread service, the metadata set for metadata of a first small file in response to receiving a first instruction instructing the second thread service to perform an operation on the first small file; and
performing, by the second thread service, the operation corresponding to the first instruction in response to locating the metadata of the first small file in the metadata set.

2. The method according to claim 1 , wherein the first instruction instructs the second thread service to read the first small file, and wherein the method further comprises returning, by the second thread service, a message that the first small file does not exist in response to the metadata of the first small file not being included in the metadata set.

3. The method according to claim 1 , wherein the first instruction instructs the second thread service to read the first small file, and wherein performing, by the second thread service, the operation corresponding to the first instruction comprises reading, by the second thread service, the metadata of the first small file.

4. The method according to claim 1 , wherein the first instruction instructs the second thread service to create the first small file, and wherein the method further comprises creating, by the second thread service, the first small file in the file system in response to the metadata of the first small file not being included in the metadata set.

5. The method according to claim 1 , wherein the first instruction instructs the second thread service to create the first small file, and wherein performing, by the second thread service, the operation corresponding to the first instruction comprises:
confirming, by the second thread service, whether to overwrite first small file content; and
returning, by the second thread service, a creation failure message in response to being unable to confirm whether to overwrite the first small file content.

6. The method according to claim 1 , wherein the first instruction instructs the second thread service to create the first small file, and wherein performing, by the second thread service, the operation corresponding to the first instruction further comprises:
confirming, by the second thread service, whether to overwrite first small file content;
deleting, by the second thread service, the metadata of the first small file in the metadata set; and
recreating, by the second thread service, metadata of the first small file in the metadata set in response to confirming that the first small file content can be overwritten.

7. The method according to claim 1 , wherein the first instruction instructs the second thread service to delete the first small file, and wherein the method further comprises deleting, by the second thread service, the first small file from the file system when the metadata of the first small file is not included in the metadata set.

8. The method according to claim 1 , wherein the first instruction instructs the second thread service to delete the first small file, and wherein performing, by the second thread service, the operation corresponding to the first instruction comprises deleting, by the second thread service, the metadata of the first small file from the metadata set in response to being permitted to delete the first file.

9. The method according to claim 8 , wherein after the second thread service deletes the metadata of the first small file in the metadata set, the method further comprises loading, by the second thread service, editlog, and deletelog into the file system, and wherein the deletelog comprises deletion information used to instruct to delete the first small file from the first file.

10. The method according to claim 1 , wherein offline merging the M small files in the file system into the one first file according to the first rule comprises:
traversing, by the first thread service, a small-file directory in the file system;
grouping, by the first thread service, all small files in the file system according to a second rule;
starting, by the first thread service, a MapReduce (MR) task to offline merge at least the M small files in each group into the one first file according to the first rule;
generating, by the first thread service, the M pieces of metadata indicating the locations of the M small files in the first file;
starting, by the first thread service, a checkpoint operation; and
loading, by the second thread service, the M pieces of metadata into the metadata set.

11. The method according to claim 9 , further comprising:
reading, by a third thread service of the distributed storage system, the deletelog and matching between the deletelog and the M pieces of metadata of the small files in the metadata set; and
deleting, by the third thread service, N small files when the deletelog instructs to delete the N small files from the first file, wherein N is a positive integer less than M; or
deleting, by the third thread service, the first file when the deletelog instructs to delete the M small files from the first file.

12. The method according to claim 1 , further comprising:
traversing, by a fourth thread service of the distributed storage system, a small-file directory in the file system and matching between the small-file directory and the metadata in the metadata set;
deleting, by the fourth thread service, the first file when a small file in the directory is not indexed to the first file; or
deleting, by invoking the second thread service, metadata of the small file in the metadata set when a name of the small file in the small-file directory and a name of the small file in the metadata set are repeated; and
clearing, by the fourth thread service, an editlog and deletelog that are generated during a deletion process.

13. The method according to claim 1 , wherein the first thread service is a thread service used to perform offline merging of small files, and wherein the second thread service is a thread service for a namespace.

14. The method according to claim 11 , wherein the third thread service is a thread service used to perform offline deleting of the small files in the first file.

15. The method according to claim 12 , wherein the fourth

Assignees

• Huawei Tech Co Ltd

Inventors

• Zhang Yong
• Cai Yicong
• Shetty S Ajith Raj

Classifications

• G06F3/0608 Primary

  Saving storage space on storage systems · CPC title

• G06F16/182

  Distributed file systems · CPC title

• G06F3/0643

  Management of files · CPC title

• G06F3/0604

  Improving or facilitating administration, e.g. storage management · CPC title

• G06F11/1435

  using file system or storage system metadata · CPC title