Data ingestion stager for time series database

Patent metadata

Field	Value
Publication number	US-10007690-B2
Application number	US-201414497521-A
Country	US
Kind code	B2
Filing date	Sep 26, 2014
Priority date	Sep 26, 2014
Publication date	Jun 26, 2018
Grant date	Jun 26, 2018

Abstract

A time series data stager that receives input data sets and outputs output data blocks for ingestion into a time series database, with the out data blocks being sent at timings according to a sliding window based on a predetermined time.

First claim

What is claimed is:

1. A method comprising:
receiving, from a set of client data driver(s) and by a time series stager system, a plurality of input data sets designed for ingestion into a time series database, with each input data set including payload data and a timestamp related to collection of the payload data;
for each given input data set of the plurality of input data sets, storing the given input data set a data storage device in the time series stager system for at least a predetermined interval of time;
for each given input data set of the plurality of input data sets, outputting the given input data set to a time series database system in an order, relative to other input data sets of the plurality of input data sets, determined by respective timestamps of the plurality of input data sets;
for each given input data set of the plurality of input data sets, responsive to outputting of the given input data set, ceasing to store the given input data set in the data storage device of the time series stager system;
receiving, from a client query driver, by the time series stager system and at a first point in time, a first query;
performing the first query against input data sets stored in the data storage device of the time series stager system at the first point in time to yield first query results; and
outputting, by the time series stager and to the client query driver, the first query results.

2. The method of claim 1 wherein:
the receipt of the plurality of input data sets occurs through an ingestion thread operating in a processor(s) set of the time series stager system.

3. The method of claim 1
the outputting of the input data sets occurs through a shipping thread operating in a processor(s) set of the time series stager system.

4. The method of claim 1 wherein the outputting of the first query results occurs through a query thread operating in a processor(s) set of the time series stager system.

5. The method of claim 1 further comprising:
maintaining an index of current input data sets of the plurality of input data sets that are currently stored in the data storage device of the time series stager system;
wherein the performance of the first query includes checking the index to determine the first query results.

6. The method of claim 1 wherein:
the data storage device includes a non-volatile data storage medium; and
the storage of the plurality of input data sets is non-volatile type data storage.

7. A computer program product (CPP) comprising:
a computer readable storage medium; and
computer code stored on the computer readable storage medium, the computer code including instructions and data for causing a processor(s) set to perform at least the following operations:
receiving, from a set of client data driver(s) and by a time series stager system, a plurality of input data sets designed for ingestion into a time series database, with each input data set including payload data and a timestamp related to collection of the payload data,
for each given input data set of the plurality of input data sets, storing the given input data set a data storage device in the time series stager system for at least a predetermined interval of time,
for each given input data set of the plurality of input data sets, outputting the given input data set to a time series database system in an order, relative to other input data sets of the plurality of input data sets, determined by respective timestamps of the plurality of input data sets,
for each given input data set of the plurality of input data sets, responsive to outputting of the given input data set, ceasing to store the given input data set in the data storage device of the time series stager system,
receiving, from a client query driver, by the time series stager system and at a first point in time, a first query,
performing the first query against input data sets stored in the data storage device of the time series stager system at the first point in time to yield first query results, and
outputting, by the time series stager and to the client query driver, the first query results.

8. The CPP of claim 7 wherein the receipt of the plurality of input data sets occurs through an ingestion thread operating in a processor(s) set of the time series stager system.

9. The CPP of claim 7 wherein the outputting of the input data sets occurs through a shipping thread operating in a processor(s) set of the time series stager system.

10. The CPP of claim 7 wherein the outputting of the first query results occurs through a query thread operating in a processor(s) set of the time series stager system.

11. The CPP of claim 7 further comprising:
maintaining an index of current input data sets of the plurality of input data sets that are currently stored in the data storage device of the time series stager system;
wherein the performance of the first query includes checking the index to determine the first query results.

12. The CPP of claim 7 wherein:
the data storage device includes a non-volatile data storage medium; and
the storage of the plurality of input data sets is non-volatile type data storage.

13. A computer system (CS) comprising:
a processor(s) set;
a computer readable storage medium; and
computer code stored on the computer readable storage medium, the computer code including instructions and data for causing the processor(s) set to perform at least the following operations:
receiving, from a set of client data driver(s) and by a time series stager system, a plurality of input data sets designed for ingestion into a time series database, with each input data set including payload data and a timestamp related to collection of the payload data,
for each given input data set of the plurality of input data sets, storing the given input data set a data storage device in the time series stager system for at least a predetermined interval of time,
for each given input data set of the plurality of input data sets, outputting the given input data set to a time series database system in an order, relative to other input data sets of the plurality of input data sets, determined by respective timestamps of the plurality of input data sets,
for each given input data set of the plurality of input data sets, responsive to outputting of the given input data set, ceasing to store the given input data set in the data storage device of the time series stager system,
receiving, from a client query driver, by the time series stager system and at a first point in time, a first query,
performing the first query against input data sets stored in the data storage device of the time series stager system at the first point in time to yield first query results, and
outputting, by the time series stager and to the client query driver, the first query results.

14. The CS of claim 13 wherein the receipt of the plurality of input data sets occurs through an ingestion thread operating in a processor(s) set of the time series stager system.

15. The CS of claim 13 wherein the outputting of the input data sets occurs through a shipping thread operating in a processor(s) set of the time series stager system.

16. The CS of claim 13 wherein the outputting of the first query results occurs through a query thread operating in a processor(s) set of the time series stager system.

17. The CS of claim 13 further comprising:
maintaining an index of current input data sets of the plurality of input data sets that are currently stored in the data storage device of the time series stager system;
wher

Assignees

• IBM

Inventors

• Finkler Ulrich A

Classifications

• G06F16/2272 Primary

  Management thereof · CPC title

• G06F16/2474

  Sequence data queries, e.g. querying versioned data · CPC title

• G06F16/258

  Data format conversion from or to a database · CPC title

• G06F17/30336 Primary

  Physics · mapped topic

• G06F17/30569

  Physics · mapped topic