System and method for data redistribution in a database

Patent metadata

Field	Value
Publication number	US-10545815-B2
Application number	US-201615227546-A
Country	US
Kind code	B2
Filing date	Aug 3, 2016
Priority date	Aug 3, 2016
Publication date	Jan 28, 2020
Grant date	Jan 28, 2020

Abstract

A method for data redistribution of a job data in a first datanode (DN) to at least one additional DN in a Massively Parallel Processing (MPP) Database (DB) is provided. The method includes recording a snapshot of the job data, creating a first data portion in the first DN and a redistribution data portion in the first DN, collecting changes to a job data copy stored in a temporary table, and initiating transfer of the redistribution data portion to the at least one additional DN.

First claim

What is claimed is:

1. A method for data redistribution of job data in a first data node (DN) to at least one additional DN in a Massively Parallel Processing (MPP) Database (DB), comprising:
recording, by a managing DN in the MPP DB, job data information;
splitting the job data into a data portion and a redistribution data portion in the first DN;
collecting changes to the job data;
identifying one or more of the collected changes to the data portion and identifying one or more of the collected changes to the redistribution data portion based on the job data information, wherein the job data information comprises information about the split of the job data;
initiating transfer of the redistribution data portion to the at least one additional DN;
merging the identified changes to the data portion into the data portion; and
merging the identified changes to the redistribution data portion into the redistribution data portion.

2. The method of claim 1 , wherein the collected changes are stored in a temporary table.

3. The method of claim 2 ,
wherein the merging the identified changes to the data portion, and the merging the identified changes to the redistribution data portion comprising:
in response to success of transferring the redistribution data portion to the at least one additional DN,
merging the identified changes to the data portion from the temporary table into the data portion in the first DN, and merging the identified changes to the redistribution data portion from the temporary table into the redistribution data portion in the at least one additional DN, and

wherein the method further comprises:
indicating the first DN can continue processing; and
indicating that the at least one additional DN is ready to begin processing.

4. The method of claim 2 , wherein the merging the identified changes to the data portion, and the merging the identified changes to the redistribution data portion comprising:
in response to failure of transferring the redistribution data portion to the at least one additional DN, merging the identified changes to the data portion from the temporary table into the data portion in the first DN, and merging the identified changes to the redistribution data portion from the temporary table into the redistribution data portion in the first DN.

5. The method of claim 4 , further comprising:
indicating that the first DN is ready to re-initiate processing in the first DN.

6. The method of claim 4 , further comprising:
re-trying the data redistribution.

7. The method of claim 1 , further comprising:
in response to failure of transferring the redistribution data portion to the at least one additional DN, re-trying the data redistribution.

8. A device for data redistribution of job data in a first data node (DN) to at least one additional DN in a Massively Parallel Processing (MPP) Database, comprising:
a non-transitory memory storage comprising instructions; and
one or more processors in communication with the memory, wherein the one or more processors execute the instructions to:
record job data information;
split the job data into a data portion and a redistribution data portion in the first DN;
collect changes to the job data;
identify one or more of the collected changes to the data portion and identify one or more of the collected changes to the redistribution data portion based on the job data information, wherein the job data information comprises information about the split of the job data;
initiate transfer of the redistribution data portion to the at least one additional DN;
merge the identified changes to the data portion into the data portion; and
merge the identified changes to the redistribution data portion into the redistribution data portion.

9. The device of claim 8 , wherein the collected changes are stored in a temporary table.

10. The device of claim 9 , wherein the one or more processors further execute the instructions to:
in response to success of transferring the redistribution data portion to the at least one additional DN, merge the identified changes to the data portion from the temporary table into the data portion in the first DN, and merge the identified changes to the redistribution data portion from the temporary table into the redistribution data portion in the at least one additional DN indicate that the first DN continues processing.

11. The device of claim 9 , the one or more processors further execute the instructions to:
in response to failure of transferring the redistribution data portion to the at least one additional DN, merge the identified changes to the data portion from the temporary table into the data portion in the first DN, and merge the identified changes to the redistribution data portion from the temporary table into the redistribution data portion in the first DN.

12. The device of claim 11 , wherein the one or more processors further execute the instructions to
indicate that the first DN is ready to re-initiate processing in the first DN.

13. The device of claim 11 , the one or more processors further execute the instructions to:
re-try the data redistribution.

14. The device of claim 8 , wherein the one or more processors further execute the instructions to:
in response to failure of transferring the redistribution data portion to the at least one additional DN, re-try the data redistribution.

Assignees

• Futurewei Technologies Inc

Inventors

• Cai Le
• Zhou Qingqing
• Sun Yang

Classifications

• G06F16/27

  Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor · CPC title

• G06F11/1451

  by selection of backup contents · CPC title

• G06F16/258

  Data format conversion from or to a database · CPC title

• G06F11/0787

  Storage of error reports, e.g. persistent data storage, storage using memory protection · CPC title

• G06F11/1469

  Backup restoration techniques · CPC title