Metadata updating method and apparatus based on columnar storage in distributed file system, and host

Patent metadata

Field	Value
Publication number	US-9678969-B2
Application number	US-201615241909-A
Country	US
Kind code	B2
Filing date	Aug 19, 2016
Priority date	Feb 24, 2014
Publication date	Jun 13, 2017
Grant date	Jun 13, 2017

Abstract

A metadata updating method based on columnar storage in a distributed file system includes acquiring to-be-updated metadata in a data table, splitting data records of the data table into multiple row groups on a row basis, converting the data table into global file metadata and multiple row group files, where the row group file includes an actual data block, a data index block, a local metadata block, a metadata index block, and a file footer, determining whether the to-be-updated metadata belongs to the global file metadata, updating local metadata when the to-be-updated metadata does not belong to the global file metadata, and adding an updated local metadata block, an updated metadata index block, and an updated file footer to the multiple row group files according to updated local metadata. Dynamic updating of metadata saves time of executing an updating operation of this type and needed computing resources.

First claim

What is claimed is:

1. A metadata updating method based on columnar storage in a distributed file system, the method comprising;
acquiring to-be-updated metadata in a data table of the distributed file system;
splitting data records of the data table into multiple row groups on a row basis;
converting the data table into global file metadata and multiple row group files, wherein the global file metadata comprise definition information of the data table, and wherein a row group file comprises an actual data block, a data index block, a local metadata block, a metadata index block, and a file footer; and
determining whether the to-be-updated metadata belongs to the global file metadata, and
wherein, when the to-be-updated metadata does not belong to the global file metadata, the method further comprises:
updating local metadata; and
adding an updated local metadata block, an updated metadata index block, and an updated file footer to the multiple row group files according to the updated local metadata.

2. The method according to claim 1 , wherein updating the local metadata comprises:
generating a map task corresponding to each row group; and
updating, using the map task, local metadata in a row group corresponding to the map task.

3. The method according to claim 2 , wherein updating the local metadata in the row group and adding the updated local metadata block, the updated metadata index block, and the updated file footer comprise:
determining a target data column related to the to-be-updated metadata; and
determining whether the to-be-updated metadata in the target data column needs to be calculated, and
wherein, when the to-be-updated metadata in the target data column does not need to be calculated, the method further comprises:
generating a new metadata block;
adding the new metadata block to a tail of the row group file;
updating the metadata index block;
adding the updated metadata index block to the tail of the row group file;
updating a file offset of the metadata index block in the file footer; and
rewriting the file footer according to the file offset of the metadata index block.

4. The method according to claim 2 , wherein updating the local metadata in the row group adding the updated local metadata block, the updated metadata index block, and the updated file footer comprise:
determining a target data column related to the to-be-updated metadata; and
determining whether the to-be-updated metadata in the target data column needs to be calculated,
wherein, when the to-be-updated metadata in the target data column needs to be calculated, the method further comprises:
calculating new metadata;
generating a new metadata block according to the calculated new metadata;
adding the new metadata block to a tail of the row group file;
updating the metadata index block;
adding the updated metadata index block to the tail of the row group file;
updating a file offset of the metadata index block in the file footer; and

rewriting the file footer according to the file offset of the metadata index block.

5. The method according to claim 4 , wherein calculating the new metadata comprises:
scanning all records of a data block that is corresponding to the target data column in the actual data block; and
performing corresponding statistical calculation.

6. The method according to claim 1 , wherein before updating the local metadata, the method comprises:
determining a location of the to-be-updated metadata; and
generating a new metadata block when the to-be-updated metadata is metadata in the multiple row group files, wherein the new metadata block comprises a corresponding addition record result.

7. The method according to claim 6 , wherein generating the new metadata block comprises:
generating a blank metadata block for a new column when the new column is added; and
inserting a corresponding addition record into the metadata block, wherein the addition record comprises a column name, an addition mark, and an optional default value.

8. The method according to claim 1 , wherein before updating the local metadata, the method comprises:
determining a location of the to-be-updated metadata; and
generating a new metadata block when the to-be-updated metadata is metadata in the multiple row group files, wherein the new metadata block comprises a corresponding deletion record result.

9. The method according to claim 7 , wherein generating the new metadata block comprises adding a corresponding deletion record to a metadata block k when the deleted k th column exists, wherein the deletion record comprises a column name and a deletion mark.

10. The method according to claim 1 , further comprising modifying a file corresponding to the global file metadata when the to-be-updated metadata belongs to the global file metadata of the data table.

11. The method of claim 1 , wherein the definition information comprises at least one a column name for each column, a data type, a primary key, a foreign key, a compression type, and a compression status.

12. The method of claim 1 , wherein the actual data block comprises actual data referring to data that a file needs to store, a user semantically needs to store, and is different from metadata.

13. The method of claim 12 , wherein the actual data are transactions.

14. A host comprising:
a processor;
a communications interface configured to acquire to-be-updated metadata in a data table of a distributed file system;
a memory configured to store instructions that when executed by the processor cause the processor to:
split data records of the data table into multiple row groups on a row basis;
convert the data table into global file metadata and multiple row group files, wherein the global file metadata comprise definition information of the data table, and wherein a row group file comprises an actual data block, a data index block, a local metadata block, a metadata index block, and a file footer;
determine whether the to-be-updated metadata belongs to the global file metadata;
update local metadata when the to-be-updated metadata does not belong to the global file metadata; and
add an updated local metadata block, an updated metadata index block, and an updated file footer to the multiple row group files according to the updated local metadata when the to-be-updated metadata does not belong to the global file metadata; and

a bus, wherein the processor, the communications interface, and the memory complete mutual communication using the bus.

15. The host according to the claim 14 , wherein the instructions further cause the processor to;
generate a map task corresponding to each row group; and
update, using the map task, local metadata in a row group corresponding to the map task.

16. The host according to the claim 15 , wherein the plurality of instructions further cause the processor to:
determine a target data column related to the to-be-updated metadata; and
determine whether the to-be-updated metadata in the target data column needs to be calculated, and
wherein when the to-be-updated metadata in the target data column does not need to be calculated, the instructions further cause the processor to:
generate a new metadata block;
add the new metadata block to a tail of the row group file;
update the metadata index block;
add the updated metadata index block to the tail of the row group file;
update a file offset of the metadata index block in the file looter; and
rewrite the file footer according to the file offset of the metadata index block.

17. The ho

Assignees

• Huawei Tech Co Ltd

Inventors

• Deng Liqun
• Zhu Junhua

Classifications

• G06F17/30194

  Physics · mapped topic

• G06F17/30185

  Physics · mapped topic

• G06F17/30238

  Physics · mapped topic

• G06F17/30091

  Physics · mapped topic

• G06F3/0638

  Organizing or formatting or addressing of data · CPC title