Managing real time data stream processing

Patent metadata

Field	Value
Publication number	US-11520796-B2
Application number	US-202016848833-A
Country	US
Kind code	B2
Filing date	Apr 14, 2020
Priority date	Apr 14, 2020
Publication date	Dec 6, 2022
Grant date	Dec 6, 2022

Abstract

A method for managing data processing includes receiving, from a user of a data query system, a data query for data stored in a data store in communication with the data query system. The method also includes receiving a staleness parameter indicating an upper time boundary for the data query. The upper time boundary limits a query response to data within the data store that is older than the upper time boundary. The method further includes determining whether the data stored within the data store satisfies the staleness parameter. When a portion of the data within the data store fails to satisfy the staleness parameter, the method includes generating the query response that excludes the portion of the data that fails to satisfy the staleness parameter.

First claim

What is claimed is:

1. A method comprising:
receiving, at data processing hardware, from a user of a data query system, a data query for data stored in a data store in communication with the data query system;
determining, by the data processing hardware, an upper time boundary for a staleness parameter, the upper time boundary limiting a query response to data within the data store that is older than the upper time boundary by:
receiving, at the data processing hardware, user data from the user;
ingesting, by the data processing hardware, the received user data into the data store to form one or more log files at a first time;
converting, by the data processing hardware, the one or more log files into a columnar data format at a second time, the columnar data format optimized for a respective query;
determining, by the data processing hardware, a time difference between the first time and the second time; and
assigning, by the data processing hardware, the time difference to the upper time boundary for the staleness parameter;

receiving, at the data processing hardware, the upper time boundary for the staleness parameter;
determining, by the data processing hardware, whether the data stored within the data store is older than the upper time boundary for the staleness parameter; and
when a portion of the data within the data store has been written to the data store more recently than the upper time boundary for the staleness parameter, generating, by the data processing hardware, the query response that excludes the portion of the data that has been written to the data store more recently than the upper time boundary of the staleness parameter.

2. The method of claim 1 , wherein receiving the staleness parameter comprises receiving the staleness parameter from the user of the data query system.

3. The method of claim 1 , further comprising:
identifying, by the data processing hardware, log files for the data stored within the data store, each log file comprising a plurality of rows of data, each row of data of the plurality of rows of data comprising a timestamp; and
determining, by the data processing hardware, a respective timestamp within the log files that most closely matches the upper time boundary of the staleness parameter; and
for the query response, reading, by the data processing hardware, data within the log files that is older than the timestamp that most closely matches the upper time boundary of the staleness parameter.

4. The method of claim 3 , wherein the timestamp indicates a time that the data store generated the respective row of data in a respective log file.

5. The method of claim 1 , further comprising:
receiving, at the data processing hardware, a set of data from the user;
ingesting, by the data processing hardware, the set of data into the data store; and
generating, by the data processing hardware, one or more log files for the set of data, each log file comprising rows of data corresponding to data from the set of data, and wherein generating the one or more log files generates a timestamp for each row of data within a respective log file.

6. A system comprising:
data processing hardware; and
memory hardware in communication with the data processing hardware, the memory hardware storing instructions that when executed on the data processing hardware cause the data processing hardware to perform operations comprising:
receiving from a user of a data query system, a data query for data stored in a data store in communication with the data query system;
determining an upper time boundary for a staleness parameter, the upper time boundary limiting a query response to data within the data store that is older than the upper time boundary by:
receiving user data from the user;
ingesting the received user data into the data store to form one or more log files at a first time;
converting the one or more log files into a columnar data format at a second time, the columnar data format optimized for a respective query;
determining a time difference between the first time and the second time; and
assigning the time difference to the upper time boundary for the staleness Parameter;

receiving the upper time boundary for the staleness parameter indicating an upper time boundary for the data query, the upper time boundary limiting a query response to data within the data store that is older than the upper time boundary;
determining whether the data stored within the data store is older than the upper time boundary for satisfies the staleness parameter; and
when a portion of the data within the data store has been written to the data store more recently than the upper time boundary for the staleness parameter, generating the query response that excludes the portion of the data that has been written to the data store more recently than the upper time boundary of the staleness parameter.

7. The system of claim 6 , wherein receiving the staleness parameter comprises receiving the staleness parameter from the user of the data query system.

8. The system of claim 6 , further comprising:
identifying log files for the data stored within the data store, each log file comprising a plurality of rows of data, each row of data of the plurality of rows of data comprising a timestamp; and
determining a respective timestamp within the log files that most closely matches the upper time boundary of the staleness parameter; and
for the query response, reading data within the log files that is older than the timestamp that most closely matches the upper time boundary of the staleness parameter.

9. The system of claim 8 , wherein the timestamp indicates a time that the data store generated the respective row of data in a respective log file.

10. The system of claim 6 , further comprising:
receiving a set of data from the user;
ingesting the set of data into the data store; and
generating one or more log files for the set of data, each log file comprising rows of data corresponding to data from the set of data, and wherein generating the one or more log files generates a timestamp for each row of data within a respective log file.

Assignees

• Google Llc

Inventors

• Edara Pavan
• Forbes Jonathan
• Yi Yang

Classifications

• G06F16/24573 Primary

  using data annotations, e.g. user-defined metadata · CPC title

• G06F16/24539

  using cached or materialised query results · CPC title

• G06F16/2477

  Temporal data queries · CPC title

• G06F16/23

  Updating · CPC title

• G06F16/248

  Presentation of query results · CPC title