System and method for data redistribution in a database

Patent metadata

Field	Value
Publication number	US-11334422-B2
Application number	US-202016737534-A
Country	US
Kind code	B2
Filing date	Jan 8, 2020
Priority date	Aug 3, 2016
Publication date	May 17, 2022
Grant date	May 17, 2022

Abstract

A method for data redistribution of a job data in a first datanode (DN) to at least one additional DN in a Massively Parallel Processing (MPP) Database (DB) is provided. The method includes recording a snapshot of the job data, creating a first data portion in the first DN and a redistribution data portion in the first DN, collecting changes to a job data copy stored in a temporary table, and initiating transfer of the redistribution data portion to the at least one additional DN.

First claim

What is claimed is:

1. A method for data redistribution of job data in a source data node (DN) to at least one destination DN in a Massively Parallel Processing (MPP) Database (DB), comprising:
recording a snapshot of the job data;
splitting the job data into a plurality of data portions, the data portions comprising a first data portion and a second data portion, and the snapshot comprising information about the split of the job data;
collecting changes to a job data copy stored in a temporary table;
identifying one or more first changes to the first data portion and identifying one or more second changes to the second data portion from the collected changes based on the snapshot;
initiating transfer of the second data portion to the at least one destination DN;
merging the identified first changes into the first data portion; and
merging the identified second changes into the second data portion.

2. The method of claim 1 , wherein the temporary table is located in a coordinator node (CN) of the MPP DB.

3. The method of claim 1 , wherein the method further comprises:
indicating that the source DN can continue processing; and
indicating that the at least one destination DN is ready to begin processing.

4. The method of claim 1 , wherein merging the identified first changes into the first data portion and merging the identified second changes into the second data portion comprises:
in response to success of transferring the second data portion to the at least one destination DN, merging the identified first changes to the first data portion from the temporary table into the first data portion in the source DN, and merging the identified second changes to the second data portion from the temporary table into the second data portion in the at least one destination DN.

5. The method of claim 1 , wherein merging the identified first changes into the first data portion and the merging the identified second changes into the second data portion comprises:
in response to failure of transferring the second data portion to the at least one destination DN, merging the identified first changes to the first data portion from the temporary table into the first data portion in the source DN, and merging the identified second changes to the second data portion from the temporary table into the second data portion in the source DN.

6. The method of claim 5 , further comprising:
indicating that the source DN is ready to re-initiate processing in the source DN.

7. The method of claim 5 , further comprising:
re-trying the data redistribution.

8. The method of claim 1 , further comprising:
in response to failure of transferring the redistribution second data portion to the at least one destination DN, re-trying the data redistribution.

9. The method of claim 1 , wherein the snapshot comprises index ranges of the job data or of a portion or portions of the job data.

10. The method of claim 1 , wherein the first data portion and the second data portion are of equal sizes.

11. A device for data redistribution of job data in a source data node (DN) to at least one destination DN in a Massively Parallel Processing (MPP) Database (DB), comprising:
a non-transitory memory storage comprising instructions; and
one or more processors in communication with the memory storage, wherein the one or more processors execute the instructions to:
record a snapshot of the job data;
split the job data into a plurality of data portions, the data portions comprising a first data portion and a second data portion, and the snapshot comprising information about the split of the job data;
collect changes to a job data copy stored in a temporary table;
identify one or more first changes to the first data portion and identify one or more second changes to the second data portion from the collected changes based on the snapshot;
initiate transfer of the second data portion to the at least one destination DN;
merge the identified first changes into the first data portion; and
merge the identified second changes into the second data portion.

12. The device of claim 11 , wherein the temporary table is located in a coordinator node (CN) of the MPP DB.

13. The device of claim 11 , wherein the one or more processors is further configured to: indicate the source DN can continue processing; and indicate that the at least one destination DN is ready to begin processing.

14. The device of claim 11 , wherein the one or more processors further execute the instructions to: in response to success of transferring the second data portion to the at least one destination DN, merging the identified first changes to the first data portion from the temporary table into the first data portion in the source DN, and merging the identified second changes to the second data portion from the temporary table into the second data portion in the at least one destination DN.

15. The device of claim 11 , wherein the one or more processors further execute the instructions to: in response to failure of transferring the second data portion to the at least one destination DN, merging the identified first changes to the first data portion from the temporary table into the first data portion in the source DN, and merging the identified second changes to the second data portion from the temporary table into the second data portion in the source DN.

16. The device of claim 15 , wherein the one or more processors further execute the instructions to: indicate that the source DN is ready to re-initiate processing in the source DN.

17. The device of claim 11 , wherein the snapshot comprises index ranges of the job data or of a portion or portions of the job data.

18. The device of claim 11 , wherein the first data portion and the second data portion are of equal sizes.

19. A non-transitory computer-readable storage medium, comprising instructions which, when executed by at least one processor, causes the processor to perform the following steps:
record a snapshot of a job data in a source data node (DN);
split the job data into a plurality of data portions, the data portions comprising a first data portion and a second data portion, and the snapshot comprising information about the split of the job data;
collect changes to a job data copy stored in a temporary table;
identify one or more first changes to the first data portion and identifying one or more second changes to the second data portion from the collected changes based on the snapshot;
initiate transfer of the second data portion to at least one destination DN;
merge the identified first changes into the first data portion; and
merge the identified second changes into the second data portion.

20. The non-transitory computer-readable storage medium of claim 19 , wherein the snapshot comprises index ranges of the job data or of a portion or portions of the job data.

Assignees

• Futurewei Technologies Inc

Inventors

• Cai Le
• Zhou Qingqing
• Sun Yang

Classifications

• G06F16/27

  Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor · CPC title

• G06F11/0793 Primary

  Remedial or corrective actions (recovery from an exception in an instruction pipeline G06F9/3861; by retry G06F11/1402; for recovering from a failure of a protocol instance or entity H04L69/40) · CPC title

• G06F2201/84

  Using snapshots, i.e. a logical point-in-time copy of the data · CPC title

• G06F16/25

  Integrating or interfacing systems involving database management systems · CPC title

• G06F2201/80

  Database-specific techniques · CPC title