Storage and compression of an aggregation file

Patent metadata

Field	Value
Publication number	US-2016124984-A1
Application number	US-201514926355-A
Country	US
Kind code	A1
Filing date	Oct 29, 2015
Priority date	Oct 30, 2014
Publication date	May 5, 2016
Grant date	—

Abstract

A method and system for storage of an aggregation file and method and system for compression of the same. The method for compressing an aggregation file includes: acquiring the aggregation file to be compressed; copying remaining files in the acquired aggregation file into a new aggregation file based on metadata of a deleted object stored in a deletion file corresponding to the acquired aggregation file; and removing the acquired aggregation file. The present invention also provides a system for compressing an aggregation file and a method and system for storing an aggregation file.

First claim

What is claimed is:

1 . A computer-implemented method for compressing an aggregation file, comprising:
acquiring the aggregation file to be compressed; copying remaining files in the acquired aggregation file into a new aggregation file based on metadata of a deleted object stored in a deletion file corresponding to the acquired aggregation file; and removing the acquired aggregation file.

2 . The computer-implemented method according to claim 1 , wherein copying the remaining files in the acquired aggregation file into the new aggregation file further comprises:
creating a temporary deletion file different from the deletion file corresponding to the acquired aggregation file.

3 . The computer-implemented method according to claim 2 , further comprising:
executing a new deletion operation in the acquired aggregation file and recording the metadata of the deleted object of the new deletion operation in the temporary deletion file in response to a new deletion operation occurring during copying of the remaining files in the acquired aggregation file into the new aggregation file.

4 . The computer-implemented method according to claim 3 , further comprising:
querying the metadata of the new aggregation file by using the metadata of temporary deletion file so as to update the metadata of the temporary deletion file; and deleting one or more objects corresponding to the metadata of temporary deletion file from the new aggregation file so that the object of the temporary deletion file corresponds to that of the new aggregation file.

5 . The computer-implemented method according to claim 4 , further comprising:
using the temporary deletion file as a new deletion file corresponding to the new aggregation file.

6 . The computer-implemented method according to claim 4 , wherein the metadata comprises at least one of a: filename of the object, offset of the object in the aggregation file and size of the object in the aggregation file.

7 . The computer-implemented method according to claim 1 , wherein acquiring the aggregation file to be compressed comprises:
calculating total deleted space of the respective aggregation file; and selecting the aggregation file having the maximum total deleted space as the aggregation file to be compressed.

8 . The computer-implemented method according to claim 1 , wherein copying the remaining files in the acquired aggregation file into the new aggregation file comprises:
determining the position(s) of the deleted objects(s) based on the deletion file corresponding to the acquired aggregation file so as to copy the aggregation file by reading sequentially one or more large blocks(s) containing a plurality of remaining objects.

9 . A system for compressing an aggregation file, comprising:
a memory; a processor communicatively coupled to the memory; and a module for compressing an aggregation file configured to carry out the steps of a method comprising: acquiring the aggregation file to be compressed; copying remaining files in the acquired aggregation file into a new aggregation file based on metadata of a deleted object stored in a deletion file corresponding to the acquired aggregation file; and removing the acquired aggregation file, thereby producing a compressed aggregation file.

10 . The system according to claim 9 , wherein the method further comprises the step of:
creating a temporary deletion file different from the deletion file of the acquired aggregation file during copying the remaining files in the acquired aggregation file into the new aggregation file.

11 . The system according to claim 10 , wherein the method further comprises the step of:
executing the new deletion operation in the acquired aggregation file and record the metadata of the deleted object of a new deletion operation in the temporary deletion file in response to the new deletion operation occurring during copying the remaining files in the acquired aggregation file into the new aggregation file.

12 . The system according to claim 11 , wherein the method further comprises the steps of:
querying the metadata of the new aggregation file by using the metadata of temporary deletion file so as to update the metadata of the temporary deletion file; and deleting one or more objects corresponding to the metadata of the temporary deletion file from the new aggregation file so that the object of the temporary deletion file corresponds to that of the new aggregation file.

13 . The system according to claim 12 , wherein the method further comprises the step of:
substituting the temporary deletion file as a new deletion file corresponding to the new aggregation file.

14 . The system according to claim 9 , wherein the metadata comprises at least one of a: file name of the object, offset and size of the object in the aggregation file, and size of the object in the aggregation file.

15 . The system according to claim 9 , wherein the acquiring step further comprises:
calculating total deleted space of respective aggregation file; and selecting the aggregation file having the maximum total deleted space as the aggregation file to be compressed.

16 . The system according to claim 9 , wherein the copying step further comprises:
determining the position(s) of the deleted objects(s) based on the deletion file corresponding to the acquired aggregation file so as to copy the aggregation file by reading sequentially one or more large blocks(s) containing a plurality of remaining objects.

17 . A computer-implemented method of storing an aggregation file, comprising:
storing a plurality of objects in an aggregation file by using an append-only technique; recording metadata corresponding to the plurality of objects of the aggregation file in an in-memory index corresponding to the aggregation file; and recording metadata corresponding to objects being deleted from the aggregation file in a specific deletion file.

18 . The computer-implemented method according to claim 17 , wherein the metadata each object in the plurality of objects comprises at least one of a: filename of the object, offset of the object in the aggregation file and size of the object in the aggregation file.

19 . A system of storing an aggregation file, the system comprising:
one or more aggregation file sections in which each section stores a plurality of objects in an aggregation file using an append-only technique; one or more in-memory index sections in which each section records metadata corresponding to the plurality of objects of the corresponding aggregation file; and one or more specific deletion file sections in which each section records metadata corresponding to objects being deleted from the corresponding aggregation file.

20 . The system according to claim 19 , wherein the metadata of each object in the plurality of obejcts comprises at least one of a: filename of the object, offset of the object in the aggregation file and size of the object in the aggregation file.

Assignees

• IBM

Inventors

• Ji Hai
• Li yuan yuan
• Yang Xiaoyang
• Zeng Chun Guang

Classifications

• G06F16/1727 Primary

  Details of free space management performed by the file system (saving storage space on storage systems G06F3/0608; management of blocks in storage devices G06F3/064) · CPC title

• G06F17/30097

  Physics · mapped topic

• G06F17/30138 Primary

  Physics · mapped topic

• G06F17/30371

  Physics · mapped topic

• G06F17/30117

  Physics · mapped topic