Page feed for efficient dataflow between distributed query engines

Patent metadata

Field	Value
Publication number	US-9836513-B2
Application number	US-201213418055-A
Country	US
Kind code	B2
Filing date	Mar 12, 2012
Priority date	Mar 12, 2012
Publication date	Dec 5, 2017
Grant date	Dec 5, 2017

Abstract

Page feed for efficient dataflow between distributed query engines is disclosed. An example method comprises storing a query result relation by a producer query as pages in a query engine buffer pool, and then sending these pages to a distributed caching platform (DCP) as the data source of another query. The example method also includes reading the result relation by a consumer query from the page stored in the DCP.

First claim

The invention claimed is:

1. A method comprising:
storing a query result relation resulting from a producer query as a local cached page in a query processor buffer pool of a first query processor, wherein the query result relation comprises a plurality of tuples returned by the producer query, wherein the plurality of tuples comprise data values from a first column of a first database table, wherein the query processor buffer pool is to store a plurality of local cached pages, wherein each of the plurality of local cached pages is a fixed-length binary page identified by a local tag;
sending, using a binary protocol, the local cached page to a distributed caching platform (DCP) to be shared as an external cached page, wherein the external cached page is visible as a data source of another query, wherein the DCP is distributed over multiple nodes, wherein the external cached page is identified by an external key that includes different data from the local tag;
reading, using the binary protocol, the external cached page by a second query processor based on the external key; and
converting the external cached page to the plurality of tuples for use by a consumer query.

2. The method of claim 1 , wherein the local tag includes a local identifier of the query result relation.

3. The method of claim 2 , wherein the local tag further includes a table space identifier, a file identifier, and a sequence number of the local cached page.

4. The method of claim 2 , wherein the external key includes a global identifier of the query result relation.

5. The method of claim 4 , wherein the external key further includes a site identifier.

6. The method of claim 1 , further comprising:
in response to sending the local cached page to the DCP:
converting the local tag to the external key, wherein the external key is included in a key-value pair; and
converting content of the local cached page to a value included in the key-value pair.

7. The method of claim 6 , further comprising storing the key-value pair in a unified hash table residing on multiple nodes.

8. The method of claim 1 , wherein reading the external cached page by a second query processor comprises using a plurality of master keys for the query result relation.

9. The method of claim 8 , wherein the plurality of master keys is composed using a known list of site identifiers for a plurality of sites.

10. A system comprising:
a first query processor including a query processor buffer pool to store a query result relation as a local cached page, wherein each local cached page stored in the query processor buffer pool is of uniform size, wherein the query result relation comprises a plurality of tuples returned by a first query, wherein each local cached page is a fixed-length binary page identified by a local tag;
a distributed caching platform (DCP) to receive, using a binary protocol, the local cached page to be shared as an external cached page, wherein the external cached page is visible as a data source of another query, wherein the DCP comprises at least one hardware processor, wherein the DCP is distributed over multiple nodes, wherein the external cached page is identified by an external key that includes different data from the local tag;
a second query processor to read, using the binary protocol, the external cached page from the DCP based on the external key, and to convert the external cached page to the plurality of tuples for use by a consumer query.

11. The system of claim 10 , wherein the local tag is composed from a local identifier of the first query result relation, a table space identifier, a file identifier, and a sequence number of the local cached page.

12. The system of claim 10 , wherein the external key comprises a global identifier of the query result relation and a site identifier.

13. The system of claim 10 , wherein each local cached page stored in the query processor buffer pool is a fixed-length binary page.

14. The system of claim 10 , wherein the DCP is further to:
convert the local tag to the external key, wherein the external key is included in a key-value pair;
convert content of the local cached page to a value included in the key-value pair; and
store the key-value pair in a unified hash table.

15. The system of claim 14 , where local cached pages of the query processor buffer pool are hash partitioned to separate portions of the unified hash table residing on separate nodes.

16. The system of claim 10 , wherein the second query processor is to read the external cached page using a plurality of master keys of the query result relation.

17. The system of claim 16 , wherein the plurality of master keys is composed using a known list of site identifiers for a plurality of sites.

18. A system comprising:
a first query processor including a query processor buffer pool to store a query result relation as a local cached page, wherein each local cached page stored in the query processor buffer pool is a fixed-length binary page identified by a local tag, wherein the query result relation comprises a plurality of tuples returned by a first query;
a distributed caching platform (DCP) to receive, using a binary protocol, the local cached page to be shared as an external cached page, wherein the external cached page is identified by an external key that includes different data from the local tag, wherein the DCP comprises at least one hardware processor;
a second query processor to read, using the binary protocol, the external cached page from the DCP based on the external key.

19. The system of claim 18 wherein the DCP is further to:
convert the local tag to the external key, wherein the external key is included in a key-value pair; and
convert content of the local cached page to a value included in the key-value pair.

20. The system of claim 19 wherein the DCP is further to store the key-value pair in a unified hash table.

Assignees

• Chen Qiming
• Hsu Meichun
• Entit Software Llc

Inventors

• Chen Qiming
• Hsu Meichun

Classifications

• G06F17/30575 Primary

  Physics · mapped topic

• G06F16/27 Primary

  Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor · CPC title

• G06F16/2471 Primary

  Distributed queries · CPC title