Data processing method and apparatus and storage medium

Patent metadata

Field	Value
Publication number	US-10698623-B2
Application number	US-201816220678-A
Country	US
Kind code	B2
Filing date	Dec 14, 2018
Priority date	Oct 8, 2016
Publication date	Jun 30, 2020
Grant date	Jun 30, 2020

Abstract

A method and an apparatus for data processing are provided. Interface circuitry connects the apparatus in a distributed system. Processing circuitry generates first hash values for a first group of virtual nodes to be added into a hash ring. The first group of virtual nodes represents a first newly added device in the distributed system. Then, the processing circuitry distributes the data in the distributed system according to the hash ring that includes the first group of virtual nodes. When the distribution of the data in the distribution system fails to satisfy a uniformity requirement, the processing circuitry identifies a specific virtual node in the first group of virtual nodes that causes the distribution to fail the uniformity requirement. Then, a hash value for a makeup virtual node to replace the specific virtual node is generated to update the distribution of the data in the distributed system.

First claim

What is claimed is:

1. A method for data processing, comprising:
generating first hash values respectively for a first group of virtual nodes to be added into a hash ring that is used to distribute data in a distributed system, the first group of virtual nodes representing a first newly added device in the distributed system;
distributing the data in the distributed system according to the hash ring that includes the first group of virtual nodes with the first hash values;
determining whether the distribution of the data in the distributed system satisfies a uniformity requirement;
identifying, when the distribution of the data in the distribution system fails to satisfy the uniformity requirement, a specific virtual node in the first group of virtual nodes that causes the distribution to fail the uniformity requirement;
generating a hash value for a makeup virtual node to replace the specific virtual node in the hash ring; and
updating the distribution of the data in the distributed system based on the hash ring with the makeup virtual node.

2. The method according to claim 1 , further comprising:
deleting, from the hash ring, the specific virtual node that causes the distribution to fail to satisfy the uniformity requirement; and
generating the hash value for the makeup virtual node that is added into the hash ring.

3. The method according to claim 1 , further comprising:
when the distribution of the data satisfies the uniformity requirement, determining whether a number of devices in the distributed system reaches a preset number; and
adding another device in the distributed system when the number of devices is lower than the preset number.

4. The method according to claim 3 , further comprising:
when the number of devices reaches the preset number, deploying the distributed system to update data storage according to the hash ring.

5. The method according to claim 4 , further comprising:
determining a piece of data for a storage update to the first newly added device according to a hash value of the piece of data and the first group of virtual nodes in the hash ring; and
storing the piece of data to the first newly added device.

6. The method according to claim 1 , further comprising:
generating second hash values respectively for a second group of virtual nodes to be added into the hash ring, the second group of virtual nodes representing a second newly added device in the distributed system; and
distributing the data in the distributed system according to the hash ring that includes the first virtual nodes with the first hash values and the second virtual nodes with the second hash values.

7. The method according to claim 1 , further comprising:
calculating a difference of data amounts of the distributed data to virtual nodes in the hash ring; and
determining that the distribution fails to satisfy the uniformity requirement when the difference exceeds a preset threshold.

8. An apparatus, comprising:
interface circuitry configured to connect the apparatus into a distributed system; and
processing circuitry configured to:
generate first hash values respectively for a first group of virtual nodes to be added into a hash ring that is used to distribute data in a distributed system, the first group of virtual nodes representing a first newly added device in the distributed system;
distribute the data in the distributed system according to the hash ring that includes the first group of virtual nodes with the first hash values;
determine whether the distribution of the data in the distributed system satisfies a uniformity requirement;
identify, when the distribution of the data in the distribution system fails to satisfy the uniformity requirement, a specific virtual node in the first group of virtual nodes that causes the distribution to fail the uniformity requirement;
generate a hash value for a makeup virtual node to replace the specific virtual node in the hash ring; and
update the distribution of the data in the distributed system based on the hash ring with the makeup virtual node.

9. The apparatus according to claim 8 , wherein the processing circuitry is configured to:
delete, from the hash ring, the specific virtual node that causes the distribution to fail to satisfy the uniformity requirement; and
generate the hash value for to the makeup virtual node that is added into the hash ring.

10. The apparatus according to claim 8 , wherein the processing circuitry is configured to:
when the distribution of the data satisfies the uniformity requirement, determine whether a number of devices in the distributed system reaches a preset number; and
add another device in the distributed system when the number of devices is lower than the preset number.

11. The apparatus according to claim 8 , wherein the processing circuitry is configured to:
generate second hash values respectively for a second group of virtual nodes to be added into the hash ring, the second group of virtual nodes representing a second newly added device in the distributed system; and
distribute the data in the distributed system according to the hash ring that includes the first virtual nodes with the first hash values and the second virtual nodes with the second hash values.

12. The apparatus according to claim 8 , wherein the processing circuitry is configured to:
calculate a difference of data amounts of the distributed data to virtual nodes in the hash ring; and
determine that the distribution fails to satisfy the uniformity requirement when the difference exceeds a preset threshold.

13. The apparatus according to claim 12 , wherein the processing circuitry is configured to:
when the number of devices reaches the preset number, deploy the distributed system to update data storage according to the hash ring.

14. The apparatus according to claim 13 , wherein:
the processing circuitry is configured to:
determine a piece of data for a storage update to the first newly added device according to a hash value of the piece of data and the first group of virtual nodes in the hash ring; and
control, via the interface circuitry, the distributed system to store the piece of data to the first newly added device.

15. A non-transitory computer-readable medium storing instructions which when executed by a computer cause the computer to perform:
generating first hash values respectively for a first group of virtual nodes to be added into a hash ring that is used to distribute data in a distributed system, the first group of virtual nodes representing a first newly added device in the distributed system;
distributing the data in the distributed system according to the hash ring that includes the first group of virtual nodes with the first hash values;
determining whether the distribution of the data in the distributed system satisfies a uniformity requirement;
identifying, when the distribution of the data in the distribution system fails to satisfy the uniformity requirement, a specific virtual node in the first group of virtual nodes that causes the distribution to fail the uniformity requirement;
generating a hash value for a makeup virtual node to replace the specific virtual node in the hash ring; and
updating the distribution of the data in the distributed system based on the hash ring with the makeup virtual node.

16. The non-transitory computer-readable medium according to claim 15 , wherein the instructions cause the computer to further perform:
deleting, from the hash ring, the specific virtual node that causes the distribution to fail to satisfy the uniformity re

Assignees

• Tencent Tech Shenzhen Co Ltd

Inventors

• Xiao Tao

Classifications

• G06F16/278 Primary

  Data partitioning, e.g. horizontal or vertical partitioning · CPC title

• G06F9/45558

  Hypervisor-specific management and integration aspects · CPC title

• H04L67/1046

  Joining mechanisms · CPC title

• H04L67/1048

  Departure or maintenance mechanisms · CPC title

• G06F3/0632

  by initialisation or re-initialisation of storage systems · CPC title