Support actual and virtual SQL dataflow by streaming infrastructure

Patent metadata

Field	Value
Publication number	US-10459920-B2
Application number	US-201214439919-A
Country	US
Kind code	B2
Filing date	Oct 31, 2012
Priority date	Oct 31, 2012
Publication date	Oct 29, 2019
Grant date	Oct 29, 2019

Abstract

A method includes binding a stationed operator to a database query engine, wherein the stationed operator comprises system utilities and an open executor to execute user-supplied application logic. The method further includes receiving an unbounded input stream of input tuples by the stationed operator, the stream of input tuples comprising a succession of chunks. The method also includes applying a query defined by the application logic to each successive chunk of the input stream to produce an output packet for each such chunk, and generating an unbounded output stream comprising the output packets.

First claim

What is claimed is:

1. A method for processing a database query on an input stream on a chunk by chunk basis, comprising:
binding a stationed operator to a database query engine, wherein:
the stationed operator is classified according to classes of stream operators, and further wherein an execution pattern of a class of the stationed operator depends on class meta-properties including input/output, blocking, caching and stream grouping characteristics,
the stationed operator comprises system utilities to connect to the database query engine and execute queries and an open executor to execute user-supplied application logic to define and apply the queries to an unbounded input stream of input tuples on the chunk by chunk basis, and
the unbounded input stream of input tuples have a stream grouping specifying tuple partitioning information for the stationed operator;

receiving the unbounded input stream of input tuples at the stationed operator on the chunk by chunk basis;
applying, by the stationed operator and based on the class of the stationed operator and the stream grouping, a query of the queries defined by the user-supplied application logic to the unbounded input stream of input tuples on the chunk by chunk basis to produce an output packet for each such chunk, wherein the query is applied based on whether the query is specified according to a database input schema;
passing a parallel execution message to a parallel stationed operator to execute multiple distributed instances of execution; and
generating an unbounded output stream comprising the output packets.

2. The method of claim 1 wherein each chunk of the input stream is defined by the user-supplied application logic as either a predefined number of input tuples or as all input tuples received during a specified period of time.

3. The method of claim 1 wherein the input tuples contain topology information, a data schema and messages for downstream stationed operators.

4. The method of claim 1 wherein the unbounded input stream of input tuples defines a table in a database.

5. The method of claim 4 wherein applying the query comprises applying the query to the table on the chunk by chunk basis, the chunk defined by the input tuples.

6. The method of claim 4 wherein generating the output stream comprises generating the output stream to comprise a message about parameters of the user-supplied application logic and the method further comprises storing results of the query in the table of the database.

7. A system for processing a database query on an input stream on a chunk by chunk basis, comprising:
a stationed operator module comprising system utilities to connect to a database query engine and execute a query and an open executor to execute user-supplied application logic to define and apply the query to an unbounded input stream of input tuples on the chunk by chunk basis, wherein:
the stationed operator is classified according to classes of stream operators, and further wherein an execution pattern of a class of the stationed operator depends on class meta-properties including input/output, blocking, caching and stream grouping characteristics, and
the unbounded input stream of input tuples have a stream grouping specifying tuple partitioning information for the stationed operator,

and further wherein the stationed operator is to:
bind to the database query engine;
receive the unbounded input stream of input tuples on the chunk by chunk basis;
apply, based on the class of the stationed operator and the stream grouping, the user-supplied application logic to each successive chunk of the unbounded input stream of input tuples on a chunk by chunk basis, wherein applying the application logic includes applying the query defined by the application logic based on whether the query is specified according to a database input schema;
produce an output packet based on the each successive chunk;
pass a parallel execution message to a parallel stationed operator to execute multiple distributed instances of execution; and
generate an unbounded output stream including the output packets.

8. The system of claim 7 wherein the unbounded input stream of input tuples defines a table in a database, and wherein the stationed operator module is to apply the user-supplied application logic to the table on the chunk by chunk basis, each such chunk defined by the input tuples, to produce the output packets.

9. The system of claim 8 wherein the stationed operator is to apply the user-supplied application logic to the table on the chunk by chunk basis to produce an output packet from each chunk.

10. The system of claim 9 wherein the stationed operator module is to output a stream of output packets that contain a result of the user-supplied application logic to each chunk.

11. The system of claim 9 wherein the stationed operator module is to generate a stream of output packets that contain a message about the parameters of the user-supplied application logic, and the stationed operator module is to store results of user-supplied application of the application logic in the database.

12. A system for processing a database query on an input stream on a chunk-wise basis, comprising:
a query engine;
a first stationed operator comprising system utilities to connect to the query engine and execute a query and an open executor to execute user-supplied application logic to define and apply the query to an unbounded input stream of input tuples on a chunk by chunk basis, and wherein the first stationed operator is further to:
receive the unbounded input stream of input tuples on the chunk by chunk basis, and to insert the input tuples of the input stream on the chunk by chunk basis into a database,
generate a first unbounded output stream of messages that contain the parameters for a next operation to be performed on the chunks of the input stream being inserted into the database on the chunk by chunk basis,
pass a parallel execution message to a parallel stationed operator to execute multiple physical instances of execution in a distributed way, and wherein:
the first stationed operator is classified according to classes of stream operators, and further wherein an execution pattern of a class of the first stationed operator depends on class meta-properties including input/output, blocking, caching and stream grouping characteristics,
the unbounded input stream of input tuples have a stream grouping specifying tuple partitioning information for the first stationed operator, and
the generating the first unbounded output stream includes applying, based on the class of the first stationed operator and the stream grouping, the query defined by the user-supplied application logic to the input stream of input tuples on the chunk by chunk basis, wherein the query is applied based on whether the query is specified according to a database input schema; and
a second stationed operator to receive the first output stream from the first stationed operator and perform the next operation defined by the messages, wherein the results of the next operation are stored in the database.

13. The system of claim 12 wherein the chunks of the input stream are defined by data packets.

14. The system of claim 12 wherein the second stationed operator is to generate a second unbounded output stream of messages that contain parameters for a further operation to be performed on the chunks of the input stream.

Assignees

• Chen Qiming
• Hsu Meichun
• Hewlett Packard Development Co

Inventors

• Chen Qiming
• Hsu Meichun

Classifications

• G06F16/951

  Indexing; Web crawling techniques · CPC title

• G06F16/25

  Integrating or interfacing systems involving database management systems · CPC title

• G06F16/24568 Primary

  Data stream processing; Continuous queries · CPC title