Flexible pivot querying of monitoring data with zero setup

Patent metadata

Field	Value
Publication number	US-9817864-B1
Application number	US-201313917586-A
Country	US
Kind code	B1
Filing date	Jun 13, 2013
Priority date	Jun 13, 2013
Publication date	Nov 14, 2017
Grant date	Nov 14, 2017

Abstract

Monitoring data can be sent, received, stored, and accessed with zero setup. Customers can send monitoring data to be stored without the customers having to perform any setup ahead-of-time, such as arbitrary lists of key-value pairs. Customers can perform zero setup pivot queries using stored monitoring data. For example, users can perform pivot queries using any available stored information, such as available metric types and dimensions, without any setup needed ahead-of-time.

First claim

We claim:

1. A method, implemented by a multi-tenant monitoring system comprising one or more computing devices, for performing a zero setup pivot query, the method comprising:
determining, by the one or more computing devices, available metric types and dimensions associated with monitoring data stored in data files, wherein the monitoring data stored in the data files is un-aggregated and is not aggregated across any dimensions;
receiving, from a customer by the one or more computing devices, a pivot query, wherein the pivot query indicates one or more metric types and one or more dimensions, wherein the one or more metric types and one or more dimensions are defined at query time, and wherein the one or more metric types and one or more dimensions are based, at query time, at least in part upon the available metric types and dimensions associated with the monitoring data;
performing, by the one or more computing devices, the pivot query using the monitoring data stored, in an un-aggregated format, in the data files according to the one or more metric types and the one or more dimensions; and
outputting results from performing the pivot query;
wherein the monitoring data is associated with the customer of the multi-tenant monitoring system, and wherein the monitoring data is grouped together by the customer within the data files; and
wherein each of the data files supports storing monitoring data for a plurality of customers of the multi-tenant monitoring system, and wherein the monitoring data stored within the data files is not indexed at a level of individual observations other than an indication of which data file contains a given customer's data.

2. The method of claim 1 wherein determining the available metric types and dimensions associated with monitoring data comprises:
reading the monitoring data for one or more time periods; and
accumulating metric types and dimensions that are available in the read monitoring data.

3. The method of claim 2 further comprising:
presenting, to the customer, an indication of at least a portion of the available metric types and dimensions that have been accumulated.

4. The method of claim 1 wherein performing the pivot query comprises:
reading, by the one or more computing devices, the monitoring data from the data files; and
aggregating the monitoring data read from the one or more data files according to the pivot query.

5. The method of claim 1 wherein performing the pivot query comprises:
for each time period of a plurality of time periods:
reading, by the one or more computing devices, a portion of the monitoring data corresponding to the time period from one or more of the data files associated with the time period.

6. The method of claim 1 wherein the pivot query is performed in linear time.

7. The method of claim 1 wherein the monitoring data stored in the data files is stored as a list of key-value pairs.

8. The method of claim 1 wherein no indexing is performed for the monitoring data stored within the data files other than the indication of which data file contains a given customer's data.

9. A multi-tenant monitoring system comprising one or more computing devices for performing a zero setup pivot query, the multi-tenant monitoring system comprising:
one or more systems comprising processing units, memory, and file storage;
the one or more systems configured to perform operations for performing a zero setup pivot query, the operations comprising:
determining available metric types and dimensions associated with monitoring data stored in data files, wherein the monitoring data stored in the data files is un-aggregated and is not aggregated across any dimensions;
receiving a pivot query, wherein the pivot query indicates one or more metric types and one or more dimensions, wherein the one or more metric types and one or more dimensions are defined at query time, and wherein the one or more metric types and one or more dimensions are based at least in part upon the available metric types and dimensions associated with the monitoring data;
performing the pivot query using the monitoring data stored, in an un-aggregated format, in the data files according to the one or more metric types and the one or more dimensions; and
outputting results from performing the pivot query;

wherein the monitoring data is associated with a customer of the multi-tenant monitoring system, and wherein the monitoring data is grouped together by the customer within the data files; and
wherein each of the data files supports storing monitoring data for a plurality of customers of the multi-tenant monitoring system, and wherein the monitoring data stored within the data files is not indexed at a level of individual observations.

10. The multi-tenant monitoring system of claim 9 wherein determining the available metric types and dimensions associated with monitoring data comprises:
reading the monitoring data for one or more time periods; and
accumulating metric types and dimensions that are available in the read monitoring data.

11. The multi-tenant monitoring system of claim 10 the operations further comprising:
presenting, to the customer, an indication of at least a portion of the available metric types and dimensions that have been accumulated.

12. The multi-tenant monitoring system of claim 9 wherein performing the pivot query comprises:
reading the monitoring data from the data files; and
aggregating the monitoring data read from the one or more data files according to the pivot query.

13. The multi-tenant monitoring system of claim 9 wherein performing the pivot query comprises:
for each time period of a plurality of time periods:
reading a portion of the monitoring data corresponding to the time period from one or more of the data files associated with the time period.

14. The multi-tenant monitoring system of claim 9 wherein the one or more systems are further configured to perform operations for receiving monitoring data supporting zero setup pivot queries, the operations comprising:
receiving, from one or more computing devices, monitoring data comprising key-value pairs; and
storing the received monitoring data in the data files.

15. The multi-tenant monitoring system of claim 14 wherein the received monitoring data is stored in the data files as lists of key-value pairs without performing any aggregation across any dimensions and not according to a schema.

16. The multi-tenant monitoring system of claim 9 wherein no indexing is performed for the monitoring data stored within the data files other than the indication of which data file contains a given customer's data.

17. A computer-readable storage medium storing computer-executable instructions for causing one or more computing devices to perform a method supporting zero setup pivot queries using monitoring data stored within a multi-tenant monitoring system, the method comprising:
determining available metric types and dimensions associated with monitoring data stored in data files, wherein the monitoring data stored in the data files is un-aggregated and is not aggregated across any dimensions;
receiving a pivot query, wherein the pivot query indicates one or more metric types and one or more dimensions, wherein the one or more metric types and one or more dimensions are defined at query time, and wherein the one or more metric types and one or more dimensions are based at least in part upon the available metric types and dimensions associated with the monitoring data;
performing the pivot query using the monitoring data store

Assignees

• Amazon Tech Inc

Inventors

• Dunagan John David
• Pai Derek Solomon
• Shim Caitlyn Rose
• Heitmann John W
• Foster Michael R

Classifications

• G06F16/2453 Primary

  Query optimisation · CPC title

• G06F17/30442 Primary

  Physics · mapped topic

• G06F16/245 Primary

  Query processing · CPC title