Maintaining data stream history for generating materialized views

Patent metadata

Field	Value
Publication number	US-2022253433-A1
Application number	US-202217722224-A
Country	US
Kind code	A1
Filing date	Apr 15, 2022
Priority date	Nov 29, 2019
Publication date	Aug 11, 2022
Grant date	—

Abstract

Materialized views may be generated at a managed materialized view platform that accepts data streams as a source. When updating a materialized view with a data stream as a source, a stored portion of the stream is used to maintain a history of changes to be made. When a join operation needs to utilize the data stream source to determine an update to the materialized view, the data stream is preserved to complete the join and update the materialized view.

First claim

What is claimed is:

1 . A system, comprising:
at least one processor; and a memory, that stores program instructions that when executed by the at least one processor cause the at least one processor to implement a materialized view management platform, configured to:
create a materialized view stored at a target data store from a first data source that is a data stream and a second data source according to a received definition for the materialized data store;
detect a maintenance event for the materialized view;
in response to detecting the maintenance event:
perform one or more queries to search a stored portion of the data stream to join at least one or more items in the stored portion with data for a change to the second data source that satisfies the definition of the materialized view;
determine a new version of the materialized view from the join of the one or more items with the data for the change; and
update the materialized view stored in the target data store to the new version of the materialized view.

2 . The system of claim 1 , wherein the stored portion is an append-only table, and wherein the materialized view management platform is further configured to:
receive one or more new items as part of the data stream; and append the one or more new items to the append-only table.

3 . The system of claim 1 , wherein to detect the maintenance event for the materialized data store, the materialized view management platform is configured to receive the change for the second data source

4 . The system of claim 1 , wherein the materialized view management platform is offered as part of a provider network and wherein the target data store is another service offered by the provider network.

5 . A method, comprising:
detecting a maintenance event for a materialized view stored at a target data store, wherein the materialized view is specified according to a definition that creates the materialized view from a first data source that is a data stream and a second data source; in response to detecting the maintenance event:
accessing a stored portion of the data stream to join at least one or more items in the stored portion with data for a change to the second data source to determine a new version of the materialized view; and
updating the materialized view stored in the target data store to the new version of the materialized view.

6 . The method of claim 5 , wherein the stored portion is an append-only table, and wherein the method further comprises:
receiving one or more new items as part of the data stream; and appending the one or more new items to the append-only table.

7 . The method of claim 6 , wherein the accessing the stored portion of the data stream to join at least one or more items in the stored portion with data for the change to the second data source to determine the new version of the materialized view is performed as part of executing a maintenance plan created for the materialized view, and wherein the method further comprises:
updating an index for performing operations in the maintenance plan to indicate the new items in the append-only table.

8 . The method of claim 7 , further comprising:
further in response to detecting the maintenance event:
based on an evaluation of the index for performing operations in the maintenance plan, rewriting the maintenance plan to include an operation to cause the accessing the stored portion of the data stream to join at least one or more items in the stored portion with data for the change to the second data source to determine the new version of the materialized view.

9 . The method of claim 5 , further comprising:
further in response to detecting the maintenance event:
accessing a stored portion of a data table that is third data source for the materialized view to join at least one or more items in the stored portion of the database with the data for the change to the second data source to determine the new version of the materialized view.

10 . The method of claim 5 , wherein the second data source is a second data stream.

11 . The method of claim 5 , further comprising:
creating the materialized view from the first data source and the second data source in the target data store according to the definition for the materialized view in response to a request to create the materialized view received at the materialized view management platform.

12 . The method of claim 11 , further comprising generating a maintenance plan for the materialized view that instructs storing the portion of the data stream for performing updates to the materialized view.

13 . The method of claim 5 , wherein detecting the maintenance event for the materialized view stored at the target data store, the program instructions cause the materialized view management platform to further implement receiving the change for the second data source

14 . One or more non-transitory, computer-readable storage media, storing program instructions that when executed on or across one or more computing devices cause the one or more computing devices to implement a materialized view management platform that implements:
detecting a maintenance event for a materialized view stored at a target data store, wherein the materialized view is specified according to a definition that creates the materialized view from a first data source that is a data stream and a second data source; in response to detecting the maintenance event:
performing one or more queries to search a stored portion of the data stream to join at least one or more items in the stored portion with data for a change to the second data source that satisfies the definition of the materialized view;
determining a new version of the materialized view from the join of the one or more items with the data for the change; and
updating the materialized view stored in the target data store to the new version of the materialized view.

15 . The one or more non-transitory, computer-readable storage media of claim 14 , wherein the one or more non-transitory, computer-readable storage media store further program instructions that when executed on or across the one or more computing devices cause the materialized view management platform to further implement:
receiving one or more new items as part of the data stream; and inserting the one or more new items into an append-only table in a database.

16 . The one or more non-transitory, computer-readable storage media of claim 15 , wherein, in performing the one or more queries to search a stored portion of the data stream, the program instructions cause the materialized view management platform to further implement:
sending the one or more queries to the database; and receiving the one or more items in a result form the database.

17 . The one or more non-transitory, computer-readable storage media of claim 15 , wherein the performing the one or more queries to search a stored portion of the data stream is performed as part of executing a maintenance plan created for the materialized view, and wherein the one or more non-transitory, computer-readable storage media store further program instructions that when executed on or across the one or more computing devices cause the materialized view management platform to further implement:
updating a data location mapping for performing operations in the maintenance plan to indicate the new items in the append-only table.

18 . The one or more non-transitory, computer-readable storage media of claim 17 , wherein the one o

Assignees

• Amazon Tech Inc

Inventors

• Deshpande Rohan
• Papakonstantinou Yannis
• Zaki Omer Ahmed
• Bharathy Karthik Gurumoorthy Subramanya
• Dugar Rajib
• Ummenthala Shivateja Reddy

Classifications

• G06F16/219

  Managing data history or versioning (querying versioned data G06F16/2474; querying temporal data G06F16/2477) · CPC title

• G06F16/24568

  Data stream processing; Continuous queries · CPC title

• G06F16/2456

  Join operations · CPC title

• G06F16/2393 Primary

  Updating materialised views · CPC title

• G06F16/2282

  Tablespace storage structures; Management thereof · CPC title