Method and apparatus for processing data in process of expanding or reducing capacity of stream computing system

Patent metadata

Field	Value
Publication number	US-11416283-B2
Application number	US-201916503145-A
Country	US
Kind code	B2
Filing date	Jul 3, 2019
Priority date	Jul 23, 2018
Publication date	Aug 16, 2022
Grant date	Aug 16, 2022

Abstract

A method and apparatus for processing stream data are provided. The method may include: acquiring a to-be-adjusted number of target execution units, the target execution unit referring to a unit executing a target program segment in a stream computing system; adjusting a number of the target execution units in the stream computing system based on the to-be-adjusted number; determining, for a target execution unit in at least one target execution unit after the adjustment, an identifier set corresponding to the target execution unit, an identifier in the identifier set being used to indicate to-be-processed data; and processing, through the target execution unit, the to-be-processed data indicated by the identifier in the corresponding identifier set.

First claim

What is claimed is:

1. A method for processing stream data, comprising:
acquiring a to-be-adjusted number of target execution units, the target execution unit referring to a unit executing a target program segment in a stream computing system;
adjusting a number of the target execution units in the stream computing system based on the to-be-adjusted number; and
determining, for a target execution unit in at least one target execution unit after the adjustment, an identifier set corresponding to the target execution unit, an identifier in the identifier set being used to indicate to-be-processed data; persisting, according to the identifier set to which the identifier of the to-be-processed data generated through running of an upstream execution unit of the target execution unit belongs, the to-be-processed data generated through the upstream execution unit of the target execution unit into a database; and processing, through the target execution unit, the to-be-processed data indicated by the identifier in the corresponding identifier set,
wherein determining, for the target execution unit in at least one target execution unit after the adjustment, the identifier set corresponding to the target execution unit, comprises: adjusting a mapping relationship between each target execution unit and each identifier set according to a preset rule, wherein a total number of identifier sets remains unchanged before and after the adjustment of the number of the target execution units in the stream computing system.

2. The method according to claim 1 , wherein after the processing, through the target execution unit, the to-be-processed data indicated by the identifier in the corresponding identifier set, the method further comprises:
sending indication information to the upstream execution unit of the target execution unit through the target execution unit, the indication information being used to indicate the to-be-processed data having been processed by the target execution unit.

3. The method according to claim 2 , wherein the processing, through the target execution unit, the to-be-processed data indicated by the identifier in the corresponding identifier set includes:
starting the at least one target execution unit after the adjustment; and
receiving and processing, through the started target execution unit, the to-be-processed data that is sent by the upstream execution unit of the target execution unit and has been determined, from the persisted to-be-processed data indicated by the identifier included in the identifier set corresponding to the target execution unit.

4. The method according to claim 1 , wherein the processing, through the target execution unit, the to-be-processed data indicated by the identifier in the corresponding identifier set includes:
de-duplicating, according to a historical record of receiving the to-be-processed data by the target execution unit in the stream computing system, the to-be-processed data sent to the target execution unit by the upstream execution unit of the target execution unit; and processing, through the target execution unit, the de-duplicated to-be-processed data indicated by the identifier in the corresponding identifier set.

5. An apparatus for processing stream data, comprising:
at least one processor; and
a memory storing instructions, the instructions when executed by the at least one processor, cause the at least one processor to perform operations, the operations comprising:
acquiring a to-be-adjusted number of target execution units, the target execution unit referring to a unit executing a target program segment in a stream computing system;
adjusting a number of the target execution units in the stream computing system based on the to-be-adjusted number; and
determining, for a target execution unit in at least one target execution unit after the adjustment, an identifier set corresponding to the target execution unit, an identifier in the identifier set being used to indicate to-be-processed data; persisting, according to the identifier set to which the identifier of the to-be-processed data generated through running of an upstream execution unit of the target execution unit belongs, the to-be-processed data generated through the upstream execution unit of the target execution unit into a database; and processing, through the target execution unit, the to-be-processed data indicated by the identifier in the corresponding identifier set,
wherein determining, for the target execution unit in at least one target execution unit after the adjustment, the identifier set corresponding to the target execution unit, comprises: adjusting a mapping relationship between each target execution unit and each identifier set according to a preset rule, wherein a total number of identifier sets remains unchanged before and after the adjustment of the number of the target execution units in the stream computing system.

6. The apparatus according to claim 5 , wherein after the processing, through the target execution unit, the to-be-processed data indicated by the identifier in the corresponding identifier set, the operations further comprise:
sending indication information to the upstream execution unit of the target execution unit through the target execution unit, the indication information being used to indicate the to-be-processed data having been processed by the target execution unit.

7. The apparatus according to claim 6 , wherein the processing, through the target execution unit, the to-be-processed data indicated by the identifier in the corresponding identifier set includes:
starting the at least one target execution unit after the adjustment; and
receiving and processing, through the started target execution unit, the to-be-processed data that is sent by the upstream execution unit of the target execution unit and has been determined, from the persisted to-be-processed data indicated by the identifier included in the identifier set corresponding to the target execution unit.

8. The apparatus according to claim 5 , wherein the processing, through the target execution unit, the to-be-processed data indicated by the identifier in the corresponding identifier set includes:
de-duplicating, according to a historical record of receiving the to-be-processed data by the target execution unit in the stream computing system, the to-be-processed data sent to the target execution unit by the upstream execution unit of the target execution unit; and
processing, through the target execution unit, the de-duplicated to-be-processed data indicated by the identifier in the corresponding identifier set.

9. A non-transitory computer readable medium, storing a computer program, wherein the computer program, when executed by a processor, causes the processor to perform operations, the operations comprising:
acquiring a to-be-adjusted number of target execution units, the target execution unit referring to a unit executing a target program segment in a stream computing system;
adjusting a number of the target execution units in the stream computing system based on the to-be-adjusted number; and
determining, for a target execution unit in at least one target execution unit after the adjustment, an identifier set corresponding to the target execution unit, an identifier in the identifier set being used to indicate to-be-processed data; persisting, according to the identifier set to which the identifier of the to-be-processed data generated through running of an upstream execution unit of the target execution unit belongs, the to-be-processed data generated through the upstream execution unit of the target execution unit into a database; and processing, through the target execution unit, the to-be-processe

Assignees

• Beijing Baidu Netcom Sci & Tech Co Ltd

Inventors

• Gao Weikang
• Wang Yanlin
• Xing Yue
• Zhang Jianwei
• Cheng Yi

Classifications

• G06F9/5077 Primary

  Logical partitioning of resources; Management or configuration of virtualized resources (specific details on emulation or internal functioning of virtual machines G06F9/455) · CPC title

• G06F9/5088

  involving task migration · CPC title

• G06F9/544

  Buffers; Shared memory; Pipes · CPC title

• G06F9/4881 Primary

  Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues · CPC title

• G06F9/4401

  Bootstrapping (security arrangements therefor G06F21/57) · CPC title