Method and apparatus for processing exploding data stream

Patent metadata

Field	Value
Publication number	US-10061858-B2
Application number	US-201514606084-A
Country	US
Kind code	B2
Filing date	Jan 27, 2015
Priority date	Feb 5, 2014
Publication date	Aug 28, 2018
Grant date	Aug 28, 2018

Abstract

Provided are a method and apparatus for processing a data stream capable of providing a data flow processing language to support real-time processing of an exploding data stream and providing an environment for executing the data flow processing language in a cluster system. The data flow-based exploding data stream processing method includes receiving a big data real-time processing service described in a real-time data flow language, interpreting the big data real-time processing service to generate a distributed stream processing service, and distributively deploying the distributed stream processing service in a cluster system including multiple nodes and configuring an execution environment for executing the distributed stream processing service in each node.

First claim

What is claimed is:

1. A computer-executable method for data flow-based exploding data stream processing in a cluster system including multiple nodes, the method comprising:
receiving a service script described in a real-time data flow language for processing a tuple-based data stream, the real-time data flow language including
a plurality of basic operators defining processing of a tuple and a tuple aggregation, and
a plurality of field operators defining processing of fields forming the tuple,

the service script including instructions corresponding to
a type of an input/output source,
a tuple unit input/output method, and
a window function defining a unit to be processed in the data stream,

by using the basic operators or by delivering a value to one of the basic operators;
interpreting the service script to configure a logical plan, which includes a plurality of logical nodes corresponding respectively to the plurality of basic operators;
optimizing the logical plan to generate a directed acyclic graph (DAG)-based distributed stream processing service;
distributively deploying the distributed stream processing service in the cluster system and configuring an execution environment for executing the distributed stream processing service in each of the multiple nodes; and
collecting state information of the nodes forming the cluster system and state information regarding a task of the distributed stream processing service processed in the nodes; and
redeploying the distributed stream processing service based on the state information.

2. The method of claim 1 , wherein the configuring of a logical plan further comprises expressing relationships between the basic operators as links of the logical nodes, and setting the field operators to a relationship in which the field operators are dependent upon the basic operators.

3. The method of claim 2 , wherein the converting into a DAG-based distributed stream processing service comprises converting the logical nodes and the links forming the logical plan into the DAG-based distributed stream processing service by using an application programming interface (API) provided from a distributed processing system.

4. The method of claim 1 , wherein the configuring of an execution environment comprises distributively deploying tasks forming the distributed stream processing service in the multiple nodes based on resource conditions of the cluster system.

5. The method of claim 4 , wherein the distributed stream processing service is a service in which a relationship between
external input/output tasks defining a task of receiving a data stream from an external source and providing the received data stream in units of window or a task of outputting a data stream to an external source

and
stream processing tasks defining a task of receiving and processing a data stream is expressed based on DAG.

6. A data flow-based exploding data stream processing system, comprising:
a hardware processor, and
a non-transitory medium having program instructions stored thereon, execution of which by the hardware processor causes the processing system to provide functions of:
a data flow language converting module configured to
interpret a service script described in a real-time data flow language, the real-time data flow language including a plurality of basic operators defining processing of a tuple and a tuple aggregation, and a plurality of field operators defining processing of fields forming the tuple, the service script including instructions corresponding to
a type of an input/output source,
a tuple unit input/output method, and
a window function defining a unit to be processed in the data stream,

by using the basic operators or by delivering a value to one of the basic operators; and
configure a logical plan that includes a plurality of logical nodes corresponding respectively to the plurality of basic operators;

a distributed stream service generating module configured to optimize the logical plan and convert the optimized logical plan into a directed acrylic graph (DAG)-based distributed stream processing service; and
a distributed stream processing service managing module configured to distributively deploy the distributed stream processing service in a cluster system including multiple nodes and configure an execution environment for executing the distributed stream processing service in each of the multiple nodes, the distributed stream processing service managing module collecting state information of the nodes forming the cluster system and state information regarding a task of the distributed stream processing service processed in the nodes, and redeploying the distributed stream processing service based on the state information.

7. The system of claim 6 , wherein the data flow language converting module further expresses relationships between the basic operators as links of the logical nodes, and sets the field operators to a relationship in which the field operators are dependent upon the basic operators.

8. The system of claim 7 , wherein the distributed stream service generating module converts the logical nodes and the links forming the logical plan into the DAG-based distributed stream processing service by using an application programming interface (API) provided from a distributed processing system.

9. The system of claim 6 , wherein the distributed stream processing service managing module distributively deploys tasks forming the distributed stream processing service in the multiple nodes based on resource conditions of the cluster system.

10. The system of claim 6 , wherein the distributed stream processing service is a service in which a relationship between
external input/output tasks defining a task of receiving a data stream from an external source and providing the received data stream in units of window or a task of outputting a data stream to an external source

and
stream processing tasks defining a task of receiving and processing a data stream is expressed based on DAG.

Assignees

• Electronics & Telecommunications Res Inst

Inventors

• Park Choon Seo
• Jeong Jin Hwan
• Lee Mi Young
• Hur Sung Jin
• Lee Myung Cheol
• Lee Yong Ju

Classifications

• G06F17/30876 Primary

  Physics · mapped topic

• G06F17/30516

  Physics · mapped topic

• G06F16/24568 Primary

  Data stream processing; Continuous queries · CPC title

• G06F16/955 Primary

  using information identifiers, e.g. uniform resource locators [URL] · CPC title