Method and System for Accelerated Stream Processing

Patent metadata

Field	Value
Publication number	US-2021218417-A1
Application number	US-202117215560-A
Country	US
Kind code	A1
Filing date	Mar 29, 2021
Priority date	May 15, 2008
Publication date	Jul 15, 2021
Grant date	—

Abstract

Disclosed herein are methods and systems for hardware-accelerating various data processing operations in a rule-based decision-making system such as a business rules engine, an event stream processor, and a complex event stream processor. Preferably, incoming data streams are checked against a plurality of rule conditions. Among the data processing operations that are hardware-accelerated include rule condition check operations, filtering operations, and path merging operations. The rule condition check operations generate rule condition check results for the processed data streams, wherein the rule condition check results are indicative of any rule conditions which have been satisfied by the data streams. The generation of such results with a low degree of latency provides enterprises with the ability to perform timely decision-making based on the data present in received data streams.

First claim

What is claimed is:

1 . A system for making a compute resource available in a network for loading a processing pipeline thereon for the compute resource to apply parallelism when processing streaming data, the streaming data comprising data arranged in a plurality of fields, the system comprising:
a processor that is addressable within a network, the processor arranged for configuration in response to a command over the network so that a processing pipeline for receiving and processing streaming data is loadable thereon; the loadable processing pipeline including a plurality of parallel paths for augmenting the streaming data with a plurality of flags indicative of a plurality of rule conditions, each of a plurality of the parallel paths including pipelined logic for performing different processing operations on the streaming data, and wherein each of a plurality of the parallel paths includes field selection logic that filters which fields of the streaming data that downstream pipelined logic in that parallel path will process, wherein a plurality of the parallel paths include field selection logic that filter for different fields of the streaming data so that the processor is thereby configurable via the loadable processing pipeline to parallel process different fields of the streaming data in different parallel paths with different processing operations.

2 . The system of claim 1 wherein the streaming data comprises a plurality of streaming events, and wherein the processing pipeline is configured to perform complex event processing on the streaming events via the parallel paths.

3 . The system of claim 2 wherein the processor comprises a field programmable gate array (FPGA) on which the processing pipeline is loadable.

4 . The system of claim 3 wherein the different processing operations comprise a plurality of different processing operations selected from the group consisting of a range check operation, a character check operation, a threshold check operation, and a matching operation.

5 . The system of claim 3 wherein at least one of the parallel paths includes range check logic for performing one of the different processing operations.

6 . The system of claim 3 wherein at least one of the parallel paths includes character check logic for performing one of the different processing operations.

7 . The system of claim 3 wherein at least one of the parallel paths includes threshold check logic for performing one of the different processing operations.

8 . The system of claim 3 wherein at least one of the parallel paths includes matching logic for performing one of the different processing operations.

9 . The system of claim 8 wherein the matching logic comprises regular expression pattern matching logic.

10 . The system of claim 3 wherein the loadable processing pipeline further comprises field parsing logic upstream from the parallel paths, wherein the field parsing logic identifies where boundaries between the fields of the streaming events are located.

11 . The system of claim 3 further comprising:
a database;
a processor that (1) manages a flow of commands and streaming events into the FPGA, including a command to load the loadable processing pipeline onto the FPGA, (2) manages a flow of processed streaming events out of the FPGA, and (3) selectively processes and loads at least a portion of the processed streaming events into the database based on the flags.

12 . The system of claim 11 further comprising:
a network interface through which the processor receives the streaming events.

13 . The system of claim 12 wherein the network interface receives the streaming events from a plurality of different data sources.

14 . The system of claim 3 further comprising:
a processor that manages a flow of commands and streaming events into the FPGA, including a command to load the loadable processing pipeline onto the FPGA.

15 . The system of claim 1 wherein the processor comprises a graphics processor unit (GPU) on which the processing pipeline is loadable.

16 . The system of claim 1 wherein the processor comprises a chip multi-processor (CMP) on which the processing pipeline is loadable.

17 . The system of claim 1 wherein the streaming data comprises a plurality of records, wherein the flags correspond to specific records and are indicative of data quality conditions for the corresponding specific records, and wherein the parallel paths augment the streaming data by adding the flags to their corresponding specific records.

18 . The system of claim 1 wherein the loadable processing pipeline further comprises join logic downstream from the parallel paths, wherein the join logic merges data from the parallel paths into a consolidated data stream.

19 . The system of claim 18 wherein the parallel paths include a bypass path that delivers the received streaming data to the join logic.

20 . A method for making a compute resource network-connectable for configuring the compute resource to apply parallelism when processing streaming data, the streaming data comprising data arranged in a plurality of fields, the method comprising:
providing a processor within a network; and providing a communication path in the network for commanding the processor to load a processing pipeline onto the processor to configure the processor for receiving and processing streaming data, the processing pipeline including a plurality of parallel paths that augment the streaming data with a plurality of flags indicative of a plurality of rule conditions, each of a plurality of the parallel paths including pipelined logic for performing different processing operations on the streaming data, and wherein each of a plurality of the parallel paths includes field selection logic that filters which fields of the streaming data that downstream pipelined logic in that parallel path will process, wherein a plurality of the parallel paths include field selection logic that filter for different fields of the streaming data so that the processor is thereby configurable via the processing pipeline to parallel process different fields of the streaming data in different parallel paths with different processing operations.

Assignees

• Ip Reservoir Llc

Inventors

• Indeck Ronald S
• Indeck David Mark
• Singla Naveen
• White Jason R

Classifications

• G06F15/7867 Primary

  with reconfigurable architecture · CPC title

• G06N5/025 Primary

  Extracting rules from data · CPC title

• G06Q10/10

  Office automation; Time management · CPC title

• G06F16/24544

  Join order optimisation · CPC title

• G06F16/24568

  Data stream processing; Continuous queries · CPC title