Performing queries to a consistent view of a data set across query engine types

Patent metadata

Field	Value
Publication number	US-11860869-B1
Application number	US-201916457154-A
Country	US
Kind code	B1
Filing date	Jun 28, 2019
Priority date	Jun 28, 2019
Publication date	Jan 2, 2024
Grant date	Jan 2, 2024

Abstract

Queries to a consistent view of a data set across different types of query engines are performed. Portions of a query received at a first type of query engine that can be performed at a different type of a query engine can be identified. A consistent view of a data set stored in a common data store can be provided by the first type of query engine to the second query engine. When the different type of query engine performs the portion of the query, the different type of query engine applies the consistent view of the data set. A result for the query can be returned by the first type of query engine based on results from the different type of query engine.

First claim

What is claimed is:

1. A system, comprising:
a first one or more processors and a first memory that implement a storage node that stores a common copy of data for a database in a common format;
a second one or more processors and a second memory that implement an Online Transaction Processing (OLTP) type of query engine for the common copy of the data of the database stored in the common format at the data node, wherein the OLTP type of query engine is communicatively coupled to the storage node via a first network connection;
a third one or more processors and a third memory that implement an Online Analytical Processing (OLAP) type of query engine for the common copy of the data of the database stored in the common format at the data node, wherein the OLAP type of query engine is communicatively coupled to the storage node over a second network connection different from the first network connection;
the OTLP type of query engine, configured to:
receive a query to the database;
identify at least a portion of the query to be performed at the OLAP type of query engine;
determine a consistent view of a data set of the database applicable to the query, the consistent view comprising one or more changes to the common copy of the data of the data set stored on the storage node;
send a request to the OLAP type of query engine to perform the portion of the query; and
provide the OLAP type of query engine with the consistent view of the data set applicable to the query;

the OLAP type of query engine, configured to:
obtain, responsive to receipt of the request to perform the portion of the query, the common copy of the data of the data set in the common format from the storage node via the second network connection,
the obtained data comprising a current value not within the consistent view of the data set;
apply the one or more changes of the consistent view of the data set provided by the OLTP query engine when evaluating the obtained data of the database as part of performing the portion of the query on the consistent view of the database; and
return a result of the portion of the query to the OLTP query engine; and

the OTLP type of query engine, further configured to:
return a final result of the query based, at least in part, on the result received from the OLAP type of query engine.

2. The system of claim 1 , wherein to identify the portion of the query to be performed at the OLAP type of query engine, the OLTP type of query engine is configured to:
generate an initial local query engine plan;
apply one or more cross-type optimization rules to evaluate the initial local query engine plan; and
generate an updated query plan that includes a plan for performing the portion of the query at the second query engine, wherein the OLTP type of query engine performs the updated query plan.

3. The system of claim 1 ,
wherein to provide the OLAP type of query engine with the consistent view of the data set identified for the query, the OLTP query engine is configured to send to the OLAP type of query engine, one or more undo log records applicable to an item of the data obtained for performing the portion the query; and
wherein to apply the consistent view of the data set provided by the OLTP type of engine when evaluating data of the database obtained from the storage node as part of performing the portion of the query the OLAP type of query engine is configured to apply the one or more undo log records to determine a different value of the item of the data to evaluate as part of performing the portion of the query.

4. The system of claim 1 , wherein the OLTP type of query engine is implemented as part of a database service of a provider network, wherein the OLAP type of query engine is implemented as part of a data warehouse service of a provider network, and wherein prior to the receipt of the query, performance of queries across the OLTP type of query engine and the OLAP type of query engine is enabled responsive to a request received via an interface implemented as part of the provider network.

5. A method, comprising:
receiving, at a first query engine, a query to a data set stored in a common format in a data store wherein the first query engine is communicatively coupled to the data store via a first network connection;
identifying, by the first query engine, at least a portion of the query to be performed at a second query engine that is a different type of query engine than the first query engine, wherein the first query engine and second query engine provide query processing for the data set stored in the common format in the data store, and wherein the second query engine is communicatively coupled to the data store over a second network connection different from the first network connection;
determining a consistent view of the data set comprising one or more changes to data of the data set stored in the data store;
providing, by the first query engine, the second query engine with the consistent view of the data set identified for the query;
applying, by the second query engine, the one or more changes of the consistent view of the data set provided by the first query engine when evaluating data obtained from the data set in the common format at the data store, via the second network connection, as part of performing the portion of the query, the obtained data comprising a current value not within the consistent view of the data set; and
returning, by the first query engine, a result of the query based, at least in part, on a result of performing the portion of the query at the second query engine.

6. The method of claim 5 ,
wherein providing the second query engine with a consistent view of the data set identified for the query comprises sending, to the second query engine, one or more undo log records applicable to an item of the data obtained for performing the portion the query; and
wherein applying the consistent view of the data set provided by the first query engine when evaluating data obtained from the data set at the data store as part of performing the portion of the query comprises applying the one or more undo log records to determine a different value of the item of the data to evaluate as part of performing the portion of the query.

7. The method of claim 5 , further comprising:
identifying, by the second query engine, metadata for performing the portion of the query; and
obtaining, by the second query engine, the metadata for performing the portion of the query from the first query engine.

8. The method of claim 5 , further comprising:
causing, by the second query engine, the data of the data set for performing the portion of the query to be obtained from the data store and reformatted into a data format supported by the second query engine before evaluating the data as part of performing the portion of the query.

9. The method of claim 5 , further comprising:
obtaining, by the second query engine, additional data for performing the portion of the query from a snapshot of the data set stored in another data store; and
evaluating, by the second query engine, the additional data as part of performing the portion of the query to include as part of the result of the portion of the query sent to the first query engine.

10. The method of claim 5 , further comprising:
accessing, by the first query engine, the data set in the data store to obtain data for performing another portion of the query; and
combining, by the first query engine, a result of performing the other portion of the query with the result of performing the portion of the query at the second query engine to generate the result of the query that is returned.

11. The method of

Assignees

• Amazon Tech Inc

Inventors

• Hwang Sangyong
• Levandoski Justin
• Yoon Yongsik
• Gupta Kamal Kant
• Krishnamurthy Saileshwar
• Pandis Ippokratis
• Petropoulos Michail

Classifications

• G06F16/24542 Primary

  Plan optimisation · CPC title

• G06F16/2379

  Updates performed during online database operations; commit processing · CPC title

• G06F16/254

  Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses · CPC title

• G06F16/283

  Multi-dimensional databases or data warehouses, e.g. MOLAP or ROLAP · CPC title

• G06F2216/03

  Data mining · CPC title