Method and apparatus for processing data

Patent metadata

Field	Value
Publication number	US-2020026553-A1
Application number	US-201916503145-A
Country	US
Kind code	A1
Filing date	Jul 3, 2019
Priority date	Jul 23, 2018
Publication date	Jan 23, 2020
Grant date	—

Abstract

Embodiments of the present disclosure disclose a method and apparatus for processing data. A specific embodiment of the method comprises: acquiring a to-be-adjusted number of target execution units, the target execution unit referring to a unit executing a target program segment in a stream computing system; adjusting a number of the target execution units in the stream computing system based on the to-be-adjusted number; determining, for a target execution unit in at least one target execution unit after the adjustment, an identifier set corresponding to the target execution unit, an identifier in the identifier set being used to indicate to-be-processed data; and processing, through the target execution unit, the to-be-processed data indicated by the identifier in the corresponding identifier set.

First claim

What is claimed is:

1 . A method for processing data, comprising:
acquiring a to-be-adjusted number of target execution units, the target execution unit referring to a unit executing a target program segment in a stream computing system; adjusting a number of the target execution units in the stream computing system based on the to-be-adjusted number; and determining, for a target execution unit in at least one target execution unit after the adjustment, an identifier set corresponding to the target execution unit, an identifier in the identifier set being used to indicate to-be-processed data; and processing, through the target execution unit, the to-be-processed data indicated by the identifier in the corresponding identifier set.

2 . The method according to claim 1 , wherein before the processing, through the target execution unit, the to-be-processed data indicated by the identifier in the corresponding identifier set, the method further comprises:
persisting, according to an identifier set to which an identifier of to-be-processed data generated through running of an upstream execution unit of the target execution unit belongs, the generated to-be-processed data through the upstream execution unit of the target execution unit.

3 . The method according to claim 2 , wherein after the processing, through the target execution unit, the to-be-processed data indicated by the identifier in the corresponding identifier set, the method further comprises:
sending indication information to the upstream execution unit of the target execution unit through the target execution unit, the indication information being used to indicate the to-be-processed data generated through the running of the upstream execution unit of the target execution unit and processed by the target execution unit.

4 . The method according to claim 2 , wherein the processing, through the target execution unit, the to-be-processed data indicated by the identifier in the corresponding identifier set includes:
restarting the at least one target execution unit after the adjustment; and receiving and processing, through the restarted target execution unit, to-be-processed data not processed by the target execution unit, wherein the to-be-processed data is sent by the upstream execution unit of the target execution unit, is in the persisted to-be-processed data indicated by the identifier included in the identifier set corresponding to the target execution unit, and is determined according to the indication information.

5 . The method according to claim 2 , wherein the processing, through the target execution unit, the to-be-processed data indicated by the identifier in the corresponding identifier set includes:
de-duplicating, according to a historical record of receiving the to-be-processed data by the target execution unit in the stream computing system, the to-be-processed data sent to the target execution unit by the upstream execution unit of the target execution unit; and processing, through the target execution unit, the de-duplicated to-be-processed data indicated by the identifier in the corresponding identifier set.

6 . An apparatus for processing data, comprising:
at least one processor; and a memory storing instructions, the instructions when executed by the at least one processor, cause the at least one processor to perform operations, the operations comprising: acquiring a to-be-adjusted number of target execution units, the target execution unit referring to a unit executing a target program segment in a stream computing system; adjusting a number of the target execution units in the stream computing system based on the to-be-adjusted number; and determining, for a target execution unit in at least one target execution unit after the adjustment, an identifier set corresponding to the target execution unit, an identifier in the identifier set being used to indicate to-be-processed data; and process, through the target execution unit, the to-be-processed data indicated by the identifier in the corresponding identifier set.

7 . The apparatus according to claim 6 , wherein before the processing, through the target execution unit, the to-be-processed data indicated by the identifier in the corresponding identifier set, the operations further comprise:
persisting, according to an identifier set to which an identifier of to-be-processed data generated through running of an upstream execution unit of the target execution unit belongs, the generated to-be-processed data through the upstream execution unit of the target execution unit.

8 . The apparatus according to claim 7 , wherein after the processing, through the target execution unit, the to-be-processed data indicated by the identifier in the corresponding identifier set, the operations further comprise:
sending indication information to the upstream execution unit of the target execution unit through the target execution unit, the indication information being used to indicate the to-be-processed data generated through the running of the upstream execution unit of the target execution unit and processed by the target execution unit.

9 . The apparatus according to claim 7 , wherein the processing, through the target execution unit, the to-be-processed data indicated by the identifier in the corresponding identifier set includes:
restarting the at least one target execution unit after the adjustment; and receiving and processing, through the restarted target execution unit, to-be-processed data not processed by the target execution unit, wherein the to-be-processed data is sent by the upstream execution unit of the target execution unit, is in the persisted to-be-processed data indicated by the identifier included in the identifier set corresponding to the target execution unit, and is determined according to the indication information.

10 . The apparatus according to claim 7 , wherein the processing, through the target execution unit, the to-be-processed data indicated by the identifier in the corresponding identifier set includes:
de-duplicating, according to a historical record of receiving the to-be-processed data by the target execution unit in the stream computing system, the to-be-processed data sent to the target execution unit by the upstream execution unit of the target execution unit; and processing, through the target execution unit, the de-duplicated to-be-processed data indicated by the identifier in the corresponding identifier set.

11 . A non-transitory computer readable medium, storing a computer program, wherein the computer program, when executed by a processor, causes the processor to perform operations, the operations comprising:
acquiring a to-be-adjusted number of target execution units, the target execution unit referring to a unit executing a target program segment in a stream computing system; adjusting a number of the target execution units in the stream computing system based on the to-be-adjusted number; and determining, for a target execution unit in at least one target execution unit after the adjustment, an identifier set corresponding to the target execution unit, an identifier in the identifier set being used to indicate to-be-processed data; and processing, through the target execution unit, the to-be-processed data indicated by the identifier in the corresponding identifier set.

Assignees

• Beijing Baidu Netcom Sci & Tec

Inventors

• Gao Weikang
• Wang Yanlin
• Xing Yue
• Zhang Jianwei
• Cheng Yi

Classifications

• G06F2009/45579

  I/O management, e.g. providing access to device drivers or storage · CPC title

• G06F9/4843 Primary

  by program, e.g. task dispatcher, supervisor, operating system · CPC title

• G06F9/4401

  Bootstrapping (security arrangements therefor G06F21/57) · CPC title

• G06F9/544

  Buffers; Shared memory; Pipes · CPC title

• G06F9/4881 Primary

  Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues · CPC title