Data processing method in stream computing system, control node, and stream computing system

Patent metadata

Field	Value
Publication number	US-10097595-B2
Application number	US-201615257722-A
Country	US
Kind code	B2
Filing date	Sep 6, 2016
Priority date	Mar 6, 2014
Publication date	Oct 9, 2018
Grant date	Oct 9, 2018

Abstract

A stream computer system and a method for processing a data stream in a stream computing system are disclosed. In an embodiment, the method includes collecting data traffic information between each working node and other working nodes and processing speed information for each working node, determining an optimized parallelism degree for each working node according to the collected data traffic information and processing speed information and adjusting a parallelism degree of the working node according to the optimized parallelism degree of the working node.

First claim

What is claimed is:

1. A method for processing a data stream in a stream computing system, wherein the stream computing system comprises a control node and multiple working nodes, the method comprising:
invoking, by the control node according to a configured parallelism degree for each working node, one or more working nodes of the multiple working nodes to process the data stream;
collecting, by the control node, data traffic information between each working node of the one or more working nodes and other working nodes, and processing speed information for each working node of the one or more working nodes;
determining, by the control node, an optimized parallelism degree for each working node of the one or more working nodes according to the collected data traffic information and processing speed information; and
determining, by the control node, whether the optimized parallelism degree for each respective working node of the one or more working nodes is the same as a parallelism degree of the respective working node, and if the optimized parallelism degree of the respective working node is not the same as the parallelism degree of the respective working node, adjusting the parallelism degree of the respective working node according to the optimized parallelism degree of the respective working node.

2. The method according to claim 1 , wherein each working node comprises one or more execution units, wherein a data stream is processed by an execution unit of a working node when the working node is invoked to process the data stream, wherein a plurality of execution units indicate a parallelism degree of a working node, and wherein adjusting the parallelism degree of the respective working node according to the optimized parallelism degree of the respective working node comprises adding, by the control node, at least one execution unit to the respective working node or deleting at least one execution unit of the respective working node according to the optimized parallelism degree of the respective working node so that a parallelism degree of the respective working node that is represented by a quantity of execution units currently comprised in the respective invoking node is the same as the optimized parallelism degree of the respective working node.

3. The method according to claim 2 , wherein adding, by the control node, at least one execution unit to the respective working node or deleting at least one execution unit of the respective working node according to the optimized parallelism degree of the respective working node comprises:
generating, by the control node, a first control instruction used to create a new execution unit, and sending the first control instruction to the respective working node so that the respective working node creates at least one new execution unit after receiving the first control instruction and creates a data channel between the new execution unit and another execution unit when the optimized parallelism degree of the respective working node is greater than the parallelism degree of the respective working node; and
generating, by the control node, a second control instruction used to delete an execution unit of the respective working node, and sending the second control instruction to the respective working node, so that the respective working node deletes at least one execution unit of the respective working node after receiving the second control instruction and deletes a data channel connected to the deleted execution unit when the optimized parallelism degree of the respective working node is less than the parallelism degree of the respective working node.

4. The method according to claim 2 , further comprising:
after adding at least one execution unit to the respective working node or deleting at least one execution unit of the respective working node according to the optimized parallelism degree of the respective working node, adjusting, by the control node according to the added or deleted at least one execution unit, a data distribution policy of an upstream working node corresponding to the respective working node, wherein the data distribution policy is used to indicate a data receiving device and a data volume of the data receiving device at a time of data receiving when the upstream working node distributes data; and
sending, by the control node, an adjusted data distribution policy to the upstream working node so that after determining a target execution unit corresponding to a downstream target working node, the upstream working node correspondingly distributes data packets to the target execution unit according to the adjusted data distribution policy.

5. The method according to claim 2 , further comprising, after adding at least one execution unit to the respective working node or deleting at least one execution unit of the respective working node according to the optimized parallelism degree of the respective working node:
adjusting, by the control node according to the added or deleted at least one execution unit, a data distribution policy of an upstream working node corresponding to the respective working node; and
sending, by the control node, an adjusted data distribution policy to the upstream working node, so that the upstream working node determines, according to the adjusted data distribution policy, a working node group to which a target working node belongs, wherein the working node group comprises at least one working node; and determines a downstream target working node from the working node group, and after determining a target execution unit corresponding to the target working node, correspondingly distributes data packets to the target execution unit.

6. A control node in a stream computing system, wherein the stream computing system comprises the control node and multiple working nodes, and wherein the control node comprises a processor, and a non-transitory memory coupled to the processor, the processor is configured to execute instructions stored in the memory so as to:
invoke, according to a configured parallelism degree for each working node, one or more working nodes of the multiple working nodes to process a data stream;
collect data traffic information between each working node of the one or more working nodes and other working nodes, and processing speed information for each working node of the one or more working nodes;
determine an optimized parallelism degree for each working node of the one or more working nodes according to the data traffic information and the processing speed information; and
determine whether the optimized parallelism degree for each respective working node of the one or more working nodes is the same as a parallelism degree of the respective working node, and when the optimized parallelism degree of the respective working node is not the same as the parallelism degree of the respective working node, adjust the parallelism degree of the respective working node according to the optimized parallelism degree of the respective working node.

7. The control node according to claim 6 , wherein each working node comprises one or more execution units, wherein a data stream is processed by an execution unit of a working node when the working node is invoked to process the data stream, wherein a plurality of execution units indicate a parallelism degree of a working node, and wherein the processor is configured to execute instructions stored in the memory so as to add at least one execution unit to the respective working node or delete at least one execution unit of the respective working node according to the optimized parallelism degree of the respective working node so that a parallelism degree of the respective working node that is represented by a quantity of execution units currently c

Assignees

• Huawei Tech Co Ltd

Inventors

• Shi Yunlong
• Xia Mingzhen
• He Cheng

Classifications

• G06F9/5083 Primary

  Techniques for rebalancing the load in a distributed system · CPC title

• H04L43/045

  for graphical visualisation of monitoring data · CPC title

• H04L65/4015 Primary

  where at least one of the additional parallel sessions is real time or time sensitive, e.g. white board sharing, collaboration or spawning of a subconference · CPC title

• H04L47/828

  Allocation of resources per group of connections, e.g. per group of users · CPC title

• G06F9/5005 Primary

  to service a request · CPC title