Datastore for aggregated measurements for metrics

Patent metadata

Field	Value
Publication number	US-10270854-B2
Application number	US-201815865043-A
Country	US
Kind code	B2
Filing date	Jan 8, 2018
Priority date	Jun 26, 2015
Publication date	Apr 23, 2019
Grant date	Apr 23, 2019

Abstract

A computing resource monitoring service receives a request to store a measurement for a metric associated with a computing resource. The request includes the measurement itself and metadata for the measurement, which specifies attributes of the measurement. Based at least in part on the metadata, the computing resource monitoring service generates a fully-qualified metric identifier and, using the identifier, selects a logical partition for placement of the measurement. From the logical partition, the computing resource monitoring service transmits the measurement to an aggregator sub-system comprising one or more in-memory datastores. The computing resource monitoring service stores the measurement in an in-memory datastore within the aggregator sub-system.

First claim

What is claimed is:

1. A computer-implemented method, comprising:
receiving a plurality of measurements and metadata specifying attributes of the plurality of measurements;
generating a fully qualified metric identifier;
serializing the plurality of measurements and the fully qualified metric identifier;
partitioning the plurality of measurements based at least in part on the fully qualified metric identifier into a plurality of logical partitions;
transmitting the plurality of measurements from the plurality of logical partitions to one or more in-memory datastores;
obtaining, from the one or more in-memory datastores, other serialized measurements for the metric;
de-serializing the plurality of measurements and the other measurements;
aggregating the de-serialized plurality of measurements with the de-serialized other measurements;
serializing the aggregated measurements; and
storing the aggregated measurements in the one or more in-memory datastores.

2. The computer-implemented method of claim 1 , further comprising:
receiving a request to retrieve the aggregated measurements, the request including the fully qualified metric identifier;
utilizing the fully qualified metric identifier to identify the one or more in-memory datastores wherein the aggregated measurements are stored;
obtaining the aggregated measurements from the one or more in-memory datastores; and
transmitting the aggregated measurements in response to the request.

3. The computer-implemented method of claim 1 , further comprising partitioning the measurements based at least in part on a timestamp.

4. The computer-implemented method of claim 1 , wherein the step of generating a fully qualified metric identifier utilizes a hash function of the metadata.

5. A non-transitory computer-readable storage medium having stored thereon executable instructions that, upon execution by one or more processors of a computer system, cause the computer system to at least:
receive, from a partition, a measurement for a metric, the measurement received in a binary serialization format, wherein the measurement includes a fully-qualified metric identifier;
identify, based at least in part on the fully-qualified metric identifier, an in-memory datastore for the measurement;
obtain, from the in-memory datastore, other measurements for the metric;
de-serialize the measurement;
aggregate the de-serialized measurement with the other measurements;
serialize the aggregated measurements; and
store the aggregated measurements in the in-memory datastore.

6. The non-transitory computer-readable storage medium of claim 5 , further comprising executable instructions that, upon execution by one or more processors of a computer system, cause the computer system to at least obtain the other measurements in a binary serialization format, de-serialize the other measurements, and aggregate the de-serialized measurement with the de-serialized other measurements.

7. The non-transitory computer-readable storage medium of claim 5 , wherein the instructions further cause the computer system to remove, based at least in part on expiration for a retention period, measurements from the in-memory datastore having a timestamp exceeding the retention period.

8. The non-transitory computer-readable storage medium of claim 5 , wherein the fully-qualified metric identifier for the measurements is generated using a hash function and metadata for the measurement, the metadata specifying attributes of the measurement.

9. The non-transitory computer-readable storage medium of claim 5 , wherein the measurement further includes a timestamp for the measurement and a unit of measurement for the measurement.

10. The non-transitory computer-readable storage medium of claim 5 , wherein at least a portion of the fully-qualified metric identifier identifies the partition of the measurement, while another portion of the metric identifier is used for unique identification of the measurement within the partition.

11. The non-transitory computer-readable storage medium of claim 5 , wherein the in-memory datastore for the measurement is identified based at least in part on the fully qualified metric identifier and a mapping of logical partitions from a metric mapping registry.

12. The non-transitory computer-readable storage medium of claim 5 , wherein the measurement is a plurality of measurements.

13. A system, comprising at least one computing device implementing one or more services, wherein the one or more services:
receive a request to store a measurement for a metric, the request including the measurement and metadata specifying attributes of the measurement;
generate, based at least in part on the metadata, a fully qualified metric identifier for the measurement;
serialize the measurement and the fully qualified metric identifier;
identify, based at least in part on the fully qualified metric identifier and a mapping of logical partitions from a metric mapping registry, an in-memory datastore for the measurement;
determine that other measurements for the metric exist in the in-memory datastore;
obtain, from the in-memory datastore, the other measurements for the metric;
de-serialize the measurement;
aggregate the de-serialized measurement with the other measurements from the in-memory datastore;
serialize the aggregated measurements; and
store the aggregated measurements in the in-memory datastore.

14. The system of claim 13 , wherein the one or more services further cause the computer device to at least obtain the other measurements in a binary serialization format, de-serialize the other measurements, and aggregate the de-serialized measurement with the de-serialized other measurements.

15. The system of claim 13 , wherein at least one of the attributes of the measurement includes a timestamp for the measurement.

16. The system of claim 13 , wherein the one or more services further:
receive a request to store a second measurement, the request including the second measurement and metadata specifying attributes of the second measurement;
generate, based at least in part on the metadata, a fully-qualified metric identifier for the second measurement;
determine, based at least in part on a timestamp for the second measurement and the fully-qualified metric identifier, that no in-memory datastores are available for placement of the second measurement as a result of the timestamp for second measurement being outside a retention period; and
store the second measurement in a queue until an in-memory datastore is available based at least in part on the timestamp and the fully-qualified metric identifier.

17. The system of claim 13 , wherein the one or more services further:
receive a request to retrieve the aggregated measurements, the request including the fully qualified metric identifier;
utilize the fully qualified metric identifier to identify the in-memory datastore wherein the aggregated measurements are stored;
obtain the aggregated measurements from the identified in-memory datastore; and
transmit the aggregated measurements in response to the request.

18. The system of claim 17 , wherein the identification of the in-memory datastore is based at least in part on a timestamp for the measurements to enable transmission of most recent measurements first.

19. The system of claim 17 , wherein aggregated measurements are replicated and stored in a plurality of in-memory datastores for the purpose of redundancy; and
wherein the one or more services generate one or more conflict reso

Assignees

• Amazon Tech Inc

Inventors

• Taylor Gary
• Dennehy Brian
• Rodriguez Sebastien Alexandre Roland
• Mangas Julio Delgado

Classifications

• H04L67/1095 Primary

  Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes · CPC title

• H04L67/1004

  Server selection for load balancing · CPC title