Architecture for metrics aggregation without service partitioning

Patent metadata

Field	Value
Publication number	US-9882830-B2
Application number	US-201514752759-A
Country	US
Kind code	B2
Filing date	Jun 26, 2015
Priority date	Jun 26, 2015
Publication date	Jan 30, 2018
Grant date	Jan 30, 2018

Abstract

A web server computer system receives a plurality of measurements for a metric from one or more computing resources associated with the web server computer system. Each measurement includes metadata for the measurement, which specifies attributes of the measurement. The web server computer system determines, for each measurement and based at least in part on the metadata, a fully qualified metric identifier for the measurement. The web server computer system uses the fully qualified metric identifier to partition the plurality of measurements into various partitions. Once completed, the web server computer system transmits a request to one or more aggregator sub-systems of a computing resource monitoring service to store the plurality of measurements.

First claim

What is claimed is:

1. A computer-implemented method, comprising:
processing incoming metrics at a web server computer system by at least, at the web server computer system:
obtaining, from a computing resource associated with a customer computer system, a plurality of measurements for a metric and including, for each measurement of the plurality of measurements, metadata specifying attributes of the measurement;
utilizing a hash function and the metadata to generate a fully qualified metric identifier for each measurement of the plurality of measurements;
obtaining, from a metric mapping registry of a computing resource monitoring service and based at least in part on a timestamp for each measurement of the plurality of measurements, a mapping of logical partitions to one or more aggregator sub-systems of the computing resource monitoring service, the one or more aggregator sub-systems for storing the plurality of measurements;
partitioning the plurality of measurements, based at least in part on the fully qualified metric identifier for each measurement and the timestamp for each measurement of the plurality of measurements, into a plurality of logical partitions, wherein at least a portion of the fully qualified metric identifier identifies a logical partition of the plurality of logical partitions, while another portion of the fully qualified metric identifier is used for unique identification of the measurement within the logical partition;
sub-partitioning the plurality of measurements in the plurality of logical partitions based at least in part on the timestamp for each measurement of the plurality of measurements;
prioritizing the plurality of measurements in the plurality of logical partitions based at least in part on the timestamp for each measurement of the plurality of measurements; and
transmitting one or more application programming interface calls to the one or more aggregator sub-systems of the computing resource monitoring service to store the plurality of measurements of the plurality of logical partitions within one or more datastores of the one or more aggregator sub-systems, the one or more application programming interface calls including the plurality of measurements.

2. The computer-implemented method of claim 1 , further comprising:
placing the plurality of measurements from the plurality of logical partitions into one or more file-based queues, the one or more file-based queues associated with the one or more aggregator sub-systems; and
delivering, from the file-based queues, the plurality of measurements to the one or more aggregator sub-systems.

3. The computer-implemented method of claim 1 , further comprising selecting the at least one measurement from the plurality of measurements based at least in part on the at least one measurement including a latest timestamp.

4. The computer-implemented method of claim 1 , further comprising:
transmitting, from the web server computer system, a request to retrieve the plurality of measurements from the one or more datastores;
determining, at a metric consolidation engine and based at least in part on the mapping of the plurality of logical partitions, the one or more aggregator sub-systems including the one or more datastores;
obtaining, from the one or more datastores, the plurality of measurements; and
providing the plurality of measurements to the web server computer system.

5. A system, comprising:
one or more processors; and
memory including instructions that, as a result of execution by the one or more processors, cause the system to:
receive a plurality of measurements for a metric, measurements of the plurality of measurements including metadata for the measurements;
determine, using a hash function and the metadata, a metric identifier for the measurements;
partition the measurements into a plurality of partitions based at least in part on the metric identifier and a timestamp for the measurements, wherein at least a portion of the metric identifier identifies a partition of the plurality of partitions, while another portion of the metric identifier is used for unique identification of the measurements within the partition; and
transmit, from the system, a message to one or more aggregator sub-systems of a computing resource monitoring service to store the measurements.

6. The system of claim 5 , wherein the instructions further cause the system to:
transmit a request to retrieve one or more measurements from one or more datastores, the request including the metric identifier and specifying a time period associated with the one or more measurements; the metric identifier and the specified time period used to identify one or more datastores wherein the one or more measurements are stored; and
obtain from the computing resource monitoring service the one or more measurements.

7. The system of claim 5 , wherein the instructions further cause the system to serialize the measurements in a binary serialization format such that the measurements in the binary serialization format include the metric identifier, a timestamp for the measurements, a unit of measurement for the measurements, and the measurements themselves.

8. The system of claim 5 , wherein the instructions further cause the system to:
obtain, based at least in part on the timestamp for the measurements, a mapping of partitions to one or more aggregator sub-systems;
identify, based at least in part on the mapping, the one or more aggregator sub-systems for the plurality of partitions; and
associate the plurality of partitions to the one or more aggregator sub-systems to enable selection of the measurements from the plurality of partitions for creation of the message.

9. The system of claim 5 , wherein the instructions further cause the system to:
determine if any partition of the plurality of partitions lacks capacity for at least one measurement of the plurality of measurements; and
purge one or more measurements having timestamps earlier than a timestamp for the at least one measurement to create capacity for placement of the at least one measurement.

10. The system of claim 5 , wherein the instructions further cause the system to:
determine, based at least in part on the metric identifier and a timestamp being outside a retention period, that no logical partitions are available for placement of a measurement; and
place the measurement in a queue until a logical partition is available for placement of the measurement.

11. The system of claim 5 , wherein the message is transmitted to the one or more aggregator sub-systems using a communications protocol configured to provide acknowledgment of successful delivery of the measurements.

12. The system of claim 5 , wherein the instructions further cause the system to:
place the measurements from the plurality of partitions into one or more queues, the one or more queues associated with the one or more aggregator sub-systems; and
deliver, from the queues, the measurements to the one or more aggregator sub-systems.

13. A non-transitory computer-readable storage medium having stored thereon executable instructions that, as a result of execution by one or more processors of a computer system, cause the computer system to at least:
obtain a measurement for a metric, the measurement including metadata specifying attributes of the measurement;
determine, using a hash function and the metadata, a metric identifier for the measurement;
partition a plurality of measurements that includes the measurement into a plurality of partitions based at least in part on the metric identifier and a timestamp for the plurality of measurements, wherein at

Assignees

• Amazon Tech Inc

Inventors

• Taylor Gary
• Dennehy Brian
• Rodriguez Sebastien Alexandre Roland
• Mangas Julio Delgado

Classifications

• H04L41/5009 Primary

  Determining service level performance parameters or violations of service level contracts, e.g. violations of agreed response time or mean time between failures [MTBF] · CPC title

• H04L47/783 Primary

  Distributed allocation of resources, e.g. bandwidth brokers · CPC title

• H04L41/5061

  characterised by the interaction between service providers and their network customers, e.g. customer relationship management · CPC title