Dynamic stream computing topology

Patent metadata

Field	Value
Publication number	US-9740745-B2
Application number	US-201414539010-A
Country	US
Kind code	B2
Filing date	Nov 12, 2014
Priority date	Nov 12, 2013
Publication date	Aug 22, 2017
Grant date	Aug 22, 2017

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for dynamic topology generation. One of the methods includes receiving, from a user, a query that specifies a search criterion and a post-processing function prototype. A topology definition is generated from the query that specifies one or more source nodes and at least one processing node, wherein the processing node is specified in the topology to output a stream of data objects corresponding to the data object identifiers and a post-processing node according to the post-processing function prototype, wherein the post-processing node is specified in the topology to receive a stream of data objects output by the processing node, and is specified in the topology to output one or more modified data objects. The topology definition is then provided to a real-time stream computing system for execution.

First claim

What is claimed is:

1. A computer-implemented method comprising:
receiving, from a user, a query that specifies a search criterion and a post-processing function prototype;
generating, from the query, a topology definition that specifies:
one or more source nodes and at least one processing node, wherein the processing node is specified in the topology to receive, from the one or more source nodes, one or more streams of data object identifiers of data objects that satisfy the search criterion, and wherein the processing node is specified in the topology to output a stream of data objects corresponding to the data object identifiers; and
a post-processing node according to the post-processing function prototype, wherein the post-processing node is specified in the topology to receive a stream of data objects output by the processing node, perform one or more operations on the data objects according to the post-processing function prototype, and is specified in the topology to output one or more modified data objects; and

providing the topology definition to a real-time stream computing system for execution.

2. The method of claim 1 , wherein the query is a search query that specifies one or more search terms, and wherein the data object identifiers of data objects that satisfy the search query are data objects having field values that satisfy the search terms.

3. The method of claim 1 , further comprising:
receiving, from the user, an argument for the post-processing function prototype, wherein the post-processing node is specified in the topology to perform the one or more operations on the data objects using the argument.

4. The method of claim 3 , wherein the post-processing function prototype is a field extraction prototype, and wherein the post-processing node generated from the post-processing function prototype is specified in the topology to extract fields from the received data objects according to the argument and provides the extracted fields as output.

5. The method of claim 3 , wherein the post-processing function prototype is a filtering prototype, and wherein the post-processing node generated from the post-processing function prototype is specified in the topology to filter the data objects received by the post-processing node according to the argument and provides the filtered data objects as output.

6. The method of claim 3 , wherein the post-processing function prototype is an aggregating prototype, and wherein the post-processing node generated from the function prototype is specified in the topology to aggregate one or more data object values specified by the argument and provides an aggregated value as output.

7. The method of claim 1 , wherein the query and the identifier of the post-processing function are text provided as input by the user.

8. The method of claim 1 , further comprising:
receiving a second identifier of a different, second post-processing function prototype,
wherein the topology definition specifies a second post-processing node according to the second post-processing function prototype, wherein the second post-processing node is specified in the topology to receive a stream of data objects output by the post-processing node, performs one or more operations on the data objects according to the second post-processing function prototype, and outputs one or more second modified data objects.

9. The method of claim 8 , wherein the query and the identifier of the post-processing function are text input received from the user, wherein the text input specifies a subscription of streams of data objects output by the post-processing node and streams of data objects received by the second post-processing node.

10. The method of claim 1 , wherein the query and the identifier of a post-processing function are received from the user at runtime.

11. The method of claim 1 , further comprising:
allocating, by the stream computing system, one or more computing nodes for each of the one or more source nodes, the processing node, and the post-processing node,
wherein the processing node subscribes to streams from the one or more source nodes, wherein the post-processing node subscribes to a stream from the processing node.

12. A system comprising:
one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:
receiving, from a user, a query that specifies a search criterion and a post-processing function prototype;
generating, from the query, a topology definition that specifies:
one or more source nodes and at least one processing node, wherein the processing node is specified in the topology to receive, from the one or more source nodes, one or more streams of data object identifiers of data objects that satisfy the search criterion, and wherein the processing node is specified in the topology to output a stream of data objects corresponding to the data object identifiers; and
a post-processing node according to the post-processing function prototype, wherein the post-processing node is specified in the topology to receive a stream of data objects output by the processing node, perform one or more operations on the data objects according to the post-processing function prototype, and is specified in the topology to output one or more modified data objects; and

providing the topology definition to a real-time stream computing system for execution.

13. The system of claim 12 , wherein the query is a search query that specifies one or more search terms, and wherein the data object identifiers of data objects that satisfy the search query are data objects having field values that satisfy the search terms.

14. The system of claim 12 , wherein the operations further comprise:
receiving, from the user, an argument for the post-processing function prototype, wherein the post-processing node is specified in the topology to perform the one or more operations on the data objects using the argument.

15. The system of claim 14 , wherein the post-processing function prototype is a field extraction prototype, and wherein the post-processing node generated from the post-processing function prototype is specified in the topology to extract fields from the received data objects according to the argument and provides the extracted fields as output.

16. The system of claim 14 , wherein the post-processing function prototype is a filtering prototype, and wherein the post-processing node generated from the post-processing function prototype is specified in the topology to filter the data objects received by the post-processing node according to the argument and provides the filtered data objects as output.

17. The system of claim 14 , wherein the post-processing function prototype is an aggregating prototype, and wherein the post-processing node generated from the function prototype is specified in the topology to aggregate one or more data object values specified by the argument and provides an aggregated value as output.

18. The system of claim 12 , wherein the query and the identifier of the post-processing function are text provided as input by the user.

19. The system of claim 12 , wherein the operations further comprise:
receiving a second identifier of a different, second post-processing function prototype,
wherein the topology definition specifies a second post-processing node according to the second post-processing function prototype, wherein the second post-processing

Assignees

• Pivotal Software Inc

Inventors

• Han Leng
• Zhao Yanhui
• Li Xiong
• Gao Xiaoming
• Li Ye

Classifications

• G06F16/24568 Primary

  Data stream processing; Continuous queries · CPC title

• G06F16/9024

  Graphs; Linked lists (G06F16/9027 takes precedence) · CPC title

• G06F17/30516 Primary

  Physics · mapped topic

• G06F17/30958

  Physics · mapped topic