Dynamic load-based merging

Patent metadata

Field	Value
Publication number	US-10523743-B2
Application number	US-201514829132-A
Country	US
Kind code	B2
Filing date	Aug 18, 2015
Priority date	Aug 27, 2014
Publication date	Dec 31, 2019
Grant date	Dec 31, 2019

Abstract

One embodiment of the present invention provides a system for facilitating dynamic load-based merging. During operation, the system identifies incremental data in memory of a first computing device. The incremental data comprises update to reference data in a storage device in a second computing device. If the size of the incremental data becomes greater than a threshold, the system selects a first server group from a plurality of server groups. The second computing device belongs to this first server group. The system then migrates data access requests from the first server group to other server groups in the plurality of server groups and merges the incremental data from the memory of the first computing device to the storage device in the second computing device. The merging includes batch sequential disk writes to the storage device of the second computing device.

First claim

What is claimed is:

1. A computer-implemented method for facilitating dynamic load-based merging, the method comprising:
storing reference data in respective storage devices in a plurality of server groups, wherein a respective server group includes a plurality of storage devices;
in response to determining that the reference data has been updated with incremental data, storing the incremental data in a memory of a first computing device distinct from the plurality of server groups, wherein the incremental data comprises an update to the reference data, which is stored in a storage device in a second computing device, and wherein the second computing device belongs to a first server group of the plurality of server groups;
in response to a size of the incremental data in the memory of the first computing device being greater than a threshold size, selecting the first server group from the plurality of server groups for merging the incremental data;
in response to selecting the first server group for merging the incremental data, migrating data access requests for the reference data from the first server group to a second server group in the plurality of server groups;
determining a traffic ratio of data access requests to the first server group with respect to data access requests to the plurality of server groups; and
in response to the traffic ratio reaching a threshold value, merging the incremental data from the memory of the first computing device with the reference data stored in the storage device in the second computing device using disk writes.

2. The method of claim 1 , wherein the reference data is divided into a plurality of tablets, wherein a tablet is a non-overlapping segment of the reference data.

3. The method of claim 2 , wherein a respective server group of the plurality of server groups stores a copy of each tablet of the plurality of tablets.

4. The method of claim 1 , wherein selecting the first server group from the plurality of server groups includes identifying the first server group based on a sequence of the plurality of server groups.

5. The method of claim 4 , further comprising:
determining whether the merging of the incremental data have been performed for a respective server group of the plurality of server groups; and
selecting, based on the sequence of the plurality of server groups, a server group for which the merging of the incremental data has not been performed.

6. The method of claim 1 , further comprising:
maintaining the traffic ratio in a data structure.

7. The method of claim 1 , further comprising:
determining whether the merging of the incremental data is complete for the first server group; and
in response to determining that the merging of the incremental data is complete for the first server group, migrating data access requests for the reference data from the second server group to the first server group.

8. The method of claim 1 , further comprising:
determining whether the first server group is needed to serve a data access request during the merging of the incremental data; and
determining resource consumption by one or more computing devices in the first server group.

9. The method of claim 8 , further comprising:
determining a performance level associated with the data access requests served from the first server group; and
in response to the performance level being below a threshold level, pausing the merging of the incremental data.

10. The method of claim 9 , further comprising releasing resources allocated for the merging of the incremental data for serving the data access requests.

11. A non-transitory storage medium storing instructions, which when executed by a processor cause the processor to perform a method for facilitating dynamic load-based merging, the method comprising:
storing reference data in respective storage devices in a plurality of server groups, wherein a respective server group includes a plurality of storage devices;
in response to determining that the reference data has been updated with incremental data, storing the incremental data in a memory of a first computing device distinct from the plurality of server groups, wherein the incremental data comprises an update to the reference data, which is stored in a storage device in a second computing device, and wherein the second computing device belongs to a first server group of the plurality of server groups;
in response to a size of the incremental data in the memory of the first computing device being greater than a threshold size, selecting the first server group from the plurality of server groups for merging the incremental data;
in response to selecting the first server group for merging the incremental data, migrating data access requests for the reference data from the first server group to a second server group in the plurality of server groups; and
determining a traffic ratio of data access requests to the first server group with respect to data access requests to the plurality of server groups; and
in response to the traffic ratio reaching a threshold value, merging the incremental data from the memory of the first computing device with the reference data stored in the storage device in the second computing device using disk writes.

12. The non-transitory storage medium of claim 11 , wherein the reference data is divided into a plurality of tablets, wherein a tablet is a non-overlapping segment of the reference data.

13. The non-transitory storage medium of claim 12 , wherein a respective server group of the plurality of server groups stores a copy of each tablet of the plurality of tablets.

14. The non-transitory storage medium of claim 11 , wherein selecting the first server group from the plurality of server groups includes identifying the first server group based on a sequence of the plurality of server groups.

15. The non-transitory storage medium of claim 14 , wherein the method further comprises:
determining whether the merging of the incremental data have been performed for a respective server group of the plurality of server groups; and
selecting, based on the sequence of the plurality of server groups, a server group for which the merging of the incremental data has not been performed.

16. The non-transitory storage medium of claim 11 , wherein the method further comprises:
maintaining the traffic ratio in a data structure.

17. The non-transitory storage medium of claim 11 , wherein the method further comprises:
determining whether the merging of the incremental data is complete for the first server group; and
in response to determining that the merging of the incremental data is complete, migrating data access requests for the reference data from the second server group to the first server group.

18. The non-transitory storage medium of claim 11 , wherein the method further comprises:
determining whether the first server group is needed to serve a data access request during the merging of the incremental data; and
determining resource consumption by one or more computing devices in the first server group.

19. The non-transitory storage medium of claim 18 , wherein the method further comprises:
determining a performance level associated with the data access requests served from the first server group; and
in response to the performance level being below a threshold level, pausing the merging of the incremental data.

20. The non-transitory storage medium of claim 19 , wherein the method further comprises releasing resources a

Assignees

• Alibaba Group Holding Ltd

Inventors

• Yang Zhenkun
• Sun Zhidong
• Xu Hu

Classifications

• H04L67/1008 Primary

  based on parameters of servers, e.g. available memory or workload (monitoring of computer activity G06F11/30) · CPC title

• G06F16/17 Primary

  Details of further file system functions · CPC title

• G06F11/3433

  for load management (allocation of a server based on load conditions G06F9/505; load rebalancing G06F9/5083; redistributing the load in a network by a load balancer H04L67/1029) · CPC title

• H04L67/1095

  Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes · CPC title

• G06Q10/06314

  Calendaring for a resource · CPC title