Metadata Updating Method and Apparatus Based on Columnar Storage in Distributed File System, and Host

Patent metadata

Field	Value
Publication number	US-2016357740-A1
Application number	US-201615241909-A
Country	US
Kind code	A1
Filing date	Aug 19, 2016
Priority date	Feb 24, 2014
Publication date	Dec 8, 2016
Grant date	—

Abstract

A metadata updating method based on columnar storage in a distributed file system includes acquiring to-be-updated metadata in a data table, splitting data records of the data table into multiple row groups on a row basis, converting the data table into global file metadata and multiple row group files, where the row group file includes an actual data block, a data index block, a local metadata block, a metadata index block, and a file footer, determining whether the to-be-updated metadata belongs to the global file metadata, updating local metadata when the to-be-updated metadata does not belong to the global file metadata, and adding an updated local metadata block, an updated metadata index block, and an updated file footer to the multiple row group files according to updated local metadata. Dynamic updating of metadata saves time of executing an updating operation of this type and needed computing resources.

First claim

What is claimed is:

1 . A metadata updating method based on columnar storage in a distributed file system, the method comprising:
acquiring to-be-updated metadata in a data table of the distributed file system; splitting data records of the data table into multiple row groups on a row basis; converting the data table into global file metadata and multiple row group files, wherein a row group file comprises an actual data block, a data index block, a local metadata block, a metadata index block, and a file footer; and determining whether the to-be-updated metadata belongs to the global file metadata, and wherein, when the to-be-updated metadata does not belong to the global file metadata, the method further comprises:
updating local metadata; and
adding an updated local metadata block, an updated metadata index block, and an updated file footer to the multiple row group files according to the updated local metadata.

2 . The method according to claim 1 , wherein updating the local metadata comprises:
generating a map task corresponding to each row group; and updating, using the map task, local metadata in a row group corresponding to the map task.

3 . The method according to claim 2 , wherein updating the local metadata in the row group and adding the updated local metadata block, the updated metadata index block, and the updated file footer comprise:
determining a target data column related to the to-be-updated metadata; and determining whether the to-be-updated metadata in the target data column needs to be calculated, and wherein, when the to-be-updated metadata in the target data column does not need to be calculated, the method further comprises:
generating a new metadata block;
adding the new metadata block to a tail of the row group file;
updating the metadata index block;
adding the updated metadata index block to the tail of the row group file;
updating a file offset of the metadata index block in the file footer; and
rewriting the file footer according to the file offset of the metadata index block.

4 . The method according to claim 2 , wherein updating the local metadata in the row group adding the updated local metadata block, the updated metadata index block, and the updated file footer comprise:
determining a target data column related to the to-be-updated metadata; and determining whether the to-be-updated metadata in the target data column needs to be calculated, wherein, when the to-be-updated metadata in the target data column needs to be calculated, the method further comprises:
calculating new metadata;
generating a new metadata block according to the calculated new metadata;
adding the new metadata block to a tail of the row group file;
updating the metadata index block;
adding the updated metadata index block to the tail of the row group file;
updating a file offset of the metadata index block in the file footer; and
rewriting the file footer according to the file offset of the metadata index block.

5 . The method according to claim 4 , wherein calculating the new metadata comprises:
scanning all records of a data block that is corresponding to the target data column in the actual data block; and performing corresponding statistical calculation.

6 . The method according to claim 1 , wherein before updating the local metadata, the method comprises:
determining a location of the to-be-updated metadata; and generating a new metadata block when the to-be-updated metadata is metadata in the multiple row group files, wherein the new metadata block comprises a corresponding addition record result.

7 . The method according to claim 1 , wherein before updating the local metadata, the method comprises:
determining a location of the to-be-updated metadata; and generating a new metadata block when the to-be-updated metadata is metadata in the multiple row group files, wherein the new metadata block comprises a corresponding deletion record result.

8 . The method according to claim 7 , wherein generating the new metadata block comprises adding a corresponding deletion record to a metadata block k when the deleted k th column exists, wherein the deletion record comprises a column name and a deletion mark.

9 . The method according to claim 6 , wherein generating the new metadata block comprises:
generating a blank metadata block for a new column when the new column is added; and inserting a corresponding addition record into the metadata block, wherein the addition record comprises a column name, an addition mark, and an optional default value.

10 . The method according to claim 1 , further comprising modifying a file corresponding to the global file metadata when the to-be-updated metadata belongs to the global file metadata of the data table.

11 . A host comprising:
a processor; a communications interface configured to acquire to-be-updated metadata in a data table of a distributed file system; a memory configured to store instructions that when executed by the processor cause the processor to:
split data records of the data table into multiple row groups on a row basis;
convert the data table into global file metadata and multiple row group files, wherein a row group file comprises an actual data block, a data index block, a local metadata block, a metadata index block, and a file footer;
determine whether the to-be-updated metadata belongs to the global file metadata;
update local metadata when the to-be-updated metadata does not belong to the global file metadata; and
add an updated local metadata block, an updated metadata index block, and an updated file footer to the multiple row group files according to the updated local metadata when the to-be-updated metadata does not belong to the global file metadata; and
a bus, wherein the processor, the communications interface, and the memory complete mutual communication using the bus.

12 . The host according to the claim 11 , wherein the instructions further cause the processor to:
generate a map task corresponding to each row group; and update, using the map task, local metadata in a row group corresponding to the map task.

13 . The host according to the claim 12 , wherein the plurality of instructions further cause the processor to:
determine a target data column related to the to-be-updated metadata; and determine whether the to-be-updated metadata in the target data column needs to be calculated, and wherein when the to-be-updated metadata in the target data column does not need to be calculated, the instructions further cause the processor to:
generate a new metadata block;
add the new metadata block to a tail of the row group file;
update the metadata index block;
add the updated metadata index block to the tail of the row group file;
update a file offset of the metadata index block in the file footer; and
rewrite the file footer according to the file offset of the metadata index block.

14 . The host according to the claim 12 , wherein the instructions further cause the processor to:
determine a target data column related to the to-be-updated metadata; and determine whether the to-be-updated metadata in the target data column needs to be calculated, and wherein when the to-be-updated metadata in the target data column needs to be calculated, the plurality of instructions further cause the processor to:
calculate new metadata;
generate a new metadata block according to the calculated new metadata;
add the new metadata block to a tail of the row group file;
update the metadata index block;
add the updated metadata index block to the tail of

Assignees

• Huawei Tech Co Ltd

Inventors

• Deng Liqun
• Zhu Junhua

Classifications

• G06F17/3012 Primary

  Physics · mapped topic

• G06F17/30091

  Physics · mapped topic

• G06F17/30194

  Physics · mapped topic

• G06F3/061

  Improving I/O performance · CPC title

• G06F16/13

  File access structures, e.g. distributed indices (arrangements of input from, or output to, record carriers G06F3/06) · CPC title