Aggregation of metrics data with fine granularity

Patent metadata

Field	Value
Publication number	US-9880919-B2
Application number	US-201514752752-A
Country	US
Kind code	B2
Filing date	Jun 26, 2015
Priority date	Jun 26, 2015
Publication date	Jan 30, 2018
Grant date	Jan 30, 2018

Abstract

A computing resource monitoring service receives a request to store metrics data. The request includes the metrics data and a fully qualified metric identifier generated based at least in part on metadata included in an earlier request to store other metrics data for the same metric. The service uses the fully qualified metric identifier to identify an aggregator datastore for storing the metrics data. Once the service has identified the aggregator datastore, the service accesses the aggregator datastore to store the metrics data such that the metrics data and the other metrics data can be obtained in response to a request to obtain data for the metric, where the request includes the metadata.

First claim

What is claimed is:

1. A computer-implemented method, comprising:
receiving, from a customer computer system, a request to store a first plurality of measurements for a metric obtained within a time period for the metric, the request including the first plurality of measurements and metadata specifying attributes of the first plurality of measurements;
utilizing, for the first plurality of measurements, a hash function and the metadata to generate a fully qualified metric identifier for the first plurality of measurements;
determining, based at least in part on the fully qualified metric identifier and a timestamp for the first plurality of measurements, one or more datastores for the first plurality of measurements;
storing the first plurality of measurements within the determined one or more datastores;
receiving, from the customer computer system, a second request to store a second plurality of measurements specifying measurements obtained within the time period for the metric, the second request including the second plurality of measurements and the fully qualified metric identifier but lacking at least a portion of the metadata;
utilizing the fully qualified metric identifier and timestamps for the second plurality of measurements to make a determination where to store the second plurality of measurements, wherein at least a portion of the fully qualified metric identifier identifies the one or more datastores where the first plurality of measurements was stored, while another portion of the metric identifier is used for unique identification of the first plurality of measurements within the one or more datastores; and
storing the second plurality of measurements according to the determination.

2. The computer-implemented method of claim 1 , further comprising:
receiving, from the customer computer system, a third request to retrieve the first plurality of measurements, the third request including the fully qualified metric identifier; and
using the fully qualified metric identifier to obtain the first plurality of measurements from the one or more datastores for the measurements.

3. The computer-implemented method of claim 1 , further comprising utilizing run length encoding to compress the first plurality of measurements and the second plurality of measurements for one or more sequential measurements having a same value.

4. The computer-implemented method of claim 1 , further comprising aggregating the first plurality of measurements with the second plurality of measurements if the determination is to store the second plurality of measurements in the same one or more datastores as the first plurality of measurements.

5. A system, comprising at least one computing device configured to implement one or more services, wherein the one or more services are configured to:
receive a request to store a plurality of measurements for a metric, the request not having the same metadata that was received with an earlier request to store other measurements for the metric but having a metric identifier generated by a hash function and metadata associated the other measurements for the metric;
determine, based on the metric identifier and a timestamp for the plurality of measurements, one or more partitions for the plurality of measurements; and
use the metric identifier to store the plurality of measurements such that information responsive to the request to retrieve measurements for the metric that specifies the metadata that was received with the earlier request includes at least both the plurality of measurements and the other measurements, wherein at least a portion of the metric identifier identifies the one or more partitions where the plurality of measurements will be stored, while another portion of the metric identifier is used for unique identification of the plurality of measurements within the one or more partitions.

6. The system of claim 5 , wherein the one or more services are further configured to aggregate the plurality of measurements with the other measurements based at least in part on the determined metric identifier.

7. The system of claim 5 , wherein use of the determined metric identifier to store the metrics data includes:
transmitting the plurality of measurements from the one or more partitions to one or more aggregator sub-system of a plurality of aggregator sub-systems for storage of the plurality of measurements; and
storing the plurality of measurements within one or more aggregator datastores of the one or more aggregator sub-systems.

8. The system of claim 7 , wherein the one or more services are further configured to:
obtain, from a metric mapping registry and based at least in part on timestamps for the measurements of the plurality of measurements, one or more mappings between the one or more partitions and the plurality of aggregator sub-systems; and
select the one or more aggregator sub-systems from the plurality of aggregator sub-systems based at least in part on the one or more mappings.

9. The system of claim 5 , wherein the one or more services are further configured to:
receive a second request to retrieve the plurality of measurements, the request specifying the metadata;
determine the metric identifier based at least in part on the metadata; and
retrieve, through use of the metric identifier and one or more time periods specified in the metadata, at least both the plurality of measurements and the other measurements.

10. The system of claim 5 , wherein the plurality of measurements are stored in a binary serialization format, the binary serialization format including the metric identifier, a timestamp for the metrics data, a unit of measurement for the metrics data, and a measurement for a metric represented by the metrics data.

11. The system of claim 5 , wherein the metric identifier is generated through use of a hash function and one or more metric attributes specified within the metadata.

12. A non-transitory computer-readable storage medium having stored thereon executable instructions that, as a result of execution by one or more processors of a computer system, cause the computer system to at least:
receive a first application programming interface call to store one or more measurements for a metric, the first application programming interface call specifying the one or more measurements and a metric identifier for the one or more measurements, the metric identifier generated using a hash function and metadata associated with other measurements for the metric included in an earlier request to store the other measurements for the metric;
utilize the metric identifier and a timestamp for the one or more measurements to identify a datastore for storing the one or more measurements, wherein at least a portion of the metric identifier identifies the datastore, while another portion of the metric identifier is used for unique identification of the one or more measurements within the datastore; and
store the one or more measurements in the identified datastore such that the one or more measurements and the other measurements can be obtained in response to a request that includes the metadata associated with the earlier request or the metric identifier for the one or more measurements.

13. The non-transitory computer-readable storage medium of claim 12 , wherein the instructions further cause the computer system to:
receive a second application programming interface call to retrieve a plurality of measurements for the metric, the second application programming interface call including the metadata;
generate, using the included metadata, the metric identifier;
identify, based at least in part on the

Assignees

• Amazon Tech Inc

Inventors

• Taylor Gary
• Dennehy Brian
• Rodriguez Sebastien Alexandre Roland
• Mangas Julio Delgado

Classifications

• G06F17/40 Primary

  Data acquisition and logging (for input to computer G06F3/00) · CPC title

• G06F11/00

  Error detection; Error correction; Monitoring (error detection, correction or monitoring in information storage based on relative movement between record carrier and transducer G11B20/18; monitoring, i.e. supervising the progress of recording or reproducing G11B27/36; in static stores G11C29/00) · CPC title

• G06F11/34 Primary

  Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation {; Recording or statistical evaluation of user activity, e.g. usability assessment} · CPC title

• G06F11/3075

  the data filtering being achieved in order to maintain consistency among the monitored data, e.g. ensuring that the monitored data belong to the same timeframe, to the same system or component · CPC title

• G06F11/3068

  where the reporting involves data format conversion · CPC title