Data processing method in stream computing system, control node, and stream computing system

Patent metadata

Field	Value
Publication number	US-10630737-B2
Application number	US-201816112236-A
Country	US
Kind code	B2
Filing date	Aug 24, 2018
Priority date	Mar 6, 2014
Publication date	Apr 21, 2020
Grant date	Apr 21, 2020

Abstract

A stream computer system and a method for processing a data stream in a stream computing system are disclosed. The method includes a first working node invokes at least one execution unit to process a data stream according to an initial parallelism degree, a control node collects information reflecting data traffic between the first working node and a second working node, and information reflecting data processing speed of the first working node, determines an optimized parallelism degree for the first working node according to the collected information, and adjusts the parallelism degree of the first working node to be consistent with the optimized parallelism degree.

First claim

What is claimed is:

1. A stream computing system, comprising:
a control node; and
a plurality of working nodes coupled to the control node,
wherein a first working node of the plurality of working nodes is configured to invoke at least one execution unit to process a data stream being input to the first working node, wherein a quantity of execution unit being invoked by the first working node to process the data stream is indicated by an initial parallelism degree of the first working node,
wherein the control node is configured to:
obtain information reflecting input data traffic and data processing speed of the first working node;
determine an optimized parallelism degree for the first working node according to the obtained information; and
adjust the initial parallelism degree of the first working node to be consistent with the optimized parallelism degree,

wherein the data stream comprises a plurality of tuples, and the optimized parallelism degree is determined according to a ratio of a tuple arrival time to a tuple processing time, and wherein the tuple arrival time and the tuple processing time are estimated according to the obtained information.

2. The stream computing system according to claim 1 , wherein the control node is further configured to adjust the initial parallelism degree of the first working node by adding at least one execution unit or deleting at least one execution unit for the first working node according to the optimized parallelism degree, and wherein a quantity of execution unit being invoked by the first working node to process the data stream after the adjustment is same as the optimized parallelism degree.

3. The stream computing system according to claim 1 , wherein the tuple arrival time reflects an average time interval at which a tuple arrives an execution unit of the first working node, and tuple processing time reflects an average time required by the execution unit to process a tuple.

4. The stream computing system according to claim 2 , wherein the control node is further configured to:
adjust a data distribution policy of a second working node according to the added or deleted at least one execution unit, wherein the data distribution policy indicates data distribution path; and
send the adjusted data distribution policy to the second working node.

5. The stream computing system according to claim 4 , wherein the initial parallelism degree of the first working node is configured in a flow graph that represents a data processing logic of the stream computing system.

6. The stream computing system according to claim 5 , wherein the second working node is an upstream working node of the first working node according to the flow graph.

7. A method for processing a data stream in a stream computing system, comprising:
invoking, by a control node, at least one execution unit to process a data stream being input to a first working node, wherein a quantity of execution unit being invoked by the first working node to process the data stream is indicated by an initial parallelism degree of the first working node, and wherein the data stream comprises a plurality of tuples;
obtaining, by the control node, information reflecting input data traffic and data processing speed of the first working node;
determining, by the control node, an optimized parallelism degree for the first working node according to a ratio of a tuple arrival time to a tuple processing time, wherein the tuple arrival time and the tuple processing time are estimated according to the obtained information; and
adjusting, by the control node, the initial parallelism degree of the first working node to be consistent with the optimized parallelism degree.

8. The method according to claim 7 , wherein adjusting the initial parallelism degree of the first working node comprises adding at least one execution unit or deleting at least one execution unit for the first working node according to the optimized parallelism degree, and a quantity of execution units invoked by the first working node to process the data stream after the adjustment is the same as the optimized parallelism degree.

9. The method according to claim 7 , wherein the tuple arrival time reflects an average time interval at which a tuple arrives an execution unit of the first working node, and tuple processing time reflects an average time required by the execution unit to process a tuple.

10. The method according to claim 8 , further comprising:
adjusting a data distribution policy of a second working node according to the added or deleted at least one execution unit, wherein the data distribution policy indicates data distribution path; and
sending the adjusted data distribution policy to the second working node.

11. The method according to claim 10 , wherein the initial parallelism degree of the first working node is configured in a flow graph that represents a data processing logic of the stream computing system.

12. The method according to claim 11 , wherein the second working node is an upstream working node of the first working node according to the flow graph.

13. A non-transitory computer readable medium including instructions, which, when executed by a processor, will cause the processor to perform the following operations:
invoking at least one execution unit to process a data stream being input to a first working node; wherein a quantity of execution unit being invoked is indicated by an initial parallelism degree of the first working node, and wherein the data stream comprises a plurality of tuples;
obtaining information reflecting input data traffic and data processing speed of the first working node;
determining an optimized parallelism degree for the first working node according to a ratio of a tuple arrival time to a tuple processing time, wherein the tuple arrival time and the tuple processing time are estimated according to the obtained information; and
adjusting the initial parallelism degree of the first working node to be consistent with the optimized parallelism degree.

14. The non-transitory computer readable medium according to claim 13 , wherein the initial parallelism degree of the first working node is adjusted by adding at least one execution unit or deleting at least one execution unit for the first working node according to the optimized parallelism degree, and a quantity of execution units invoked to process the data stream after the adjustment is the same as the optimized parallelism degree.

15. The non-transitory computer readable medium according to claim 13 , wherein the tuple arrival time reflects an average time interval at which a tuple arrives an execution unit of the first working node, and tuple processing time reflects an average time required by the execution unit to process a tuple.

16. The non-transitory computer readable medium according to claim 13 , wherein the initial parallelism degree of the first working node is configured in a flow graph that represents a data processing logic.

Assignees

• Huawei Tech Co Ltd

Inventors

• Shi Yunlong
• Xia Mingzhen
• He Cheng

Classifications

• H04L47/828

  Allocation of resources per group of connections, e.g. per group of users · CPC title

• G06F9/5083 Primary

  Techniques for rebalancing the load in a distributed system · CPC title

• H04L43/045

  for graphical visualisation of monitoring data · CPC title

• H04L65/4015 Primary

  where at least one of the additional parallel sessions is real time or time sensitive, e.g. white board sharing, collaboration or spawning of a subconference · CPC title

• G06F9/5005 Primary

  to service a request · CPC title