System for embedding stream processing execution in a database

Patent metadata

Field	Value
Publication number	US-12314264-B2
Application number	US-202117402388-A
Country	US
Kind code	B2
Filing date	Aug 13, 2021
Priority date	Feb 15, 2019
Publication date	May 27, 2025
Grant date	May 27, 2025

Abstract

A database management system, comprising: a storage adapted for storing: a plurality of data objects organized according to a data model, and a plurality of stream operator wrappers each wrapping a stream operator and having at least one port for receiving, via a network, instructions for: scheduling activation of the wrapped stream operator, connecting the wrapped stream operator with another stream operator wrapped by another of the plurality of stream operator wrappers, and/or deploying the wrapped stream operator; a processing circuitry for executing the plurality of stream operator wrappers.

First claim

What is claimed is:

1. A database management system, comprising:
a memory having the following stored thereon:
a plurality of data objects organized according to a data model, and
a plurality of stream operator wrappers, wherein each of the plurality of stream operator wrappers wraps a stream operator and has at least one port, wherein the at least one port is configured to receive, via a network, instructions from a remote stream engine, wherein the instructions are determined by the remote stream engine based on a plurality of messages from the plurality of stream operator wrappers, and wherein the instructions comprise at least one of scheduling activation of the wrapped stream operator, connecting the wrapped stream operator with another stream operator wrapped by another of the plurality of stream operator wrappers, and deploying the wrapped stream operator, and wherein the at least one port comprises at least one output port for outputting data to the remote stream engine, wherein the remote stream engine schedules operation of a plurality of stream operators wrapped in the plurality of stream operator wrappers; and

a processor configured to execute the plurality of stream operator wrappers, wherein the processor is configured to execute at least two of the plurality of stream operator wrappers simultaneously for processing a data flow of a common thread.

2. The database management system of claim 1 , wherein each of the plurality of stream operator wrappers includes an entry point for receiving a data flow from another of the plurality of stream operator wrappers which is interconnected thereto.

3. The database management system of claim 1 , wherein the stream operator is a member of a group consisting of the following operators: map, flatmap, reduce, join, window, filter, processfunction, groupby, keyby, shuffle, group, iterate, match, and aggregate.

4. The database management system of claim 1 , wherein the database management system is part of a distributed stream processing pipeline.

5. The database management system of claim 1 , wherein the processor executes each of at least two of the plurality of stream operator wrappers for combining a logic defined by the respective stream operator and at least two of the plurality of data objects.

6. The database management system of claim 1 , wherein the instructions are received for implementing a stream processing pipeline integration.

7. The database management system of claim 1 , wherein the instructions are received to control a job execution for enabling coordination and distribution of jobs according to a stream processing plan.

8. The database management system of claim 7 , wherein the stream processing plan causes the processor to build a stream application topology.

9. The database management system of claim 1 , wherein the stream operator is a single thread application defining logic for processing data from the plurality of data objects.

10. The database management system of claim 1 , wherein the stream operator is a user defined function.

11. The database management system of claim 1 , wherein the at least one port comprises at least one output port for outputting data to the remote stream engine, wherein the remote stream engine supervises the execution of the respective stream operator.

12. A stream engine, comprising:
an interface in communication, via a network, with a remote database management system, wherein the interface is configured to receive a plurality of messages from a plurality of stream operator wrappers executed on the remote database management system, wherein the remote database management system comprises a memory having a plurality of data objects and the plurality of stream operator wrappers stored thereon, and wherein each of the plurality of messages is indicative of an outcome of executing a stream operator wrapped by one of the plurality of stream operator wrappers, and wherein the stream engine schedules operation of a plurality of stream operators wrapped in the plurality of stream operator wrappers; and
a processor configured to execute a policy for scheduling or interconnecting at least some of a plurality of stream operators wrapped in the plurality of stream operator wrappers according to data in the plurality of messages, and to send instructions to the remote database management system based on the executed policy, wherein the processor is configured to execute at least two of the plurality of stream operator wrappers simultaneously for processing a data flow of a common thread.

13. The stream engine of claim 12 , wherein the processor is configured to instruct a deployment of one or more of the plurality of stream operators by sending an indication to one or more of the plurality of stream operator wrappers via the network.

14. A system, comprising:
a stream engine; and
a database management sub-system comprising a processor and a memory having the following stored thereon:
a plurality of data objects organized according to a data model; and
a plurality of stream operator wrappers, wherein each of the plurality of stream operator wrappers wraps a stream operator and has at least one port;

wherein at least two of the plurality of stream operator wrappers are executed by the processor based on instructions, wherein the instructions are received through the at least one port from the stream engine via a network, wherein the instructions are determined by the stream engine based on a plurality of messages from the plurality of stream operator wrappers, and wherein the instructions comprise at least one of scheduling activation of the wrapped stream operator, connecting the wrapped stream operator with another stream operator wrapped by another of the plurality of stream operator wrappers, and deploying the wrapped stream operator, and wherein the at least one port comprises at least one output port for outputting data to the stream engine, wherein the stream engine schedules operation of a plurality of stream operators wrapped in the plurality of stream operator wrappers, wherein the processor is configured to execute at least two of the plurality of stream operator wrappers simultaneously for processing a data flow of a common thread.

15. The system of claim 14 , wherein each of the plurality of stream operator wrappers includes an entry point for receiving a data flow from another of the plurality of stream operator wrappers which is interconnected thereto.

16. The system of claim 14 , wherein the stream operator is a member of a group consisting of the following operators: map, flatmap, reduce, join, window, filter, processfunction, groupby, keyby, shuffle, group, iterate, match, and aggregate.

17. The system of claim 14 , wherein the database management sub-system is part of a distributed stream processing pipeline.

Assignees

• Huawei Cloud Computing Tech Co Ltd

Inventors

• Tudoran Radu
• Nozdrin Alexander
• Bortoli Stefano
• Hassan Mohamad Al Hajj
• Axenie Cristian
• Li Hailin
• Brasche Goetz

Classifications

• G06F16/2457

  with adaptation to user needs · CPC title

• G06F16/2456

  Join operations · CPC title

• G06F16/244

  Grouping and aggregation · CPC title

• G06F16/24568 Primary

  Data stream processing; Continuous queries · CPC title