Methods and systems that efficiently store metric data to enable period and peak detection

Patent metadata

Field	Value
Publication number	US-10592169-B2
Application number	US-201715822612-A
Country	US
Kind code	B2
Filing date	Nov 27, 2017
Priority date	Nov 27, 2017
Publication date	Mar 17, 2020
Grant date	Mar 17, 2020

Abstract

The current document is directed to methods and systems that collect metric data within computing facilities, including large data centers and cloud-computing facilities. In a described implementation, input metric data is compressed by replacing each metric data point with a one-bit, two-bit, four-bit, or eight-bit compressed data value. During a first time window following reception of a metric data point, the metric data point remains available in uncompressed form to facilitate data analysis and monitoring functionalities that use uncompressed metric data. During a second time window, the metric data point is compressed and stored in memory, where the compressed data point remains available for data analysis and monitoring functionalities that use compressed metric data for detection of peaks, periodic patterns, and other characteristics. Finally, the compressed data point is archived in mass storage, where it remains available to data-analysis and management functionalities for a lengthy time period.

First claim

The invention claimed is:

1. A metric-data collection-and-storage subsystem within a distributed computer system, the metric-data collection-and-storage subsystem comprising:
one or more processors;
one or more memories;
one or more data-storage devices;
one or more virtual machines instantiated by computer instructions stored in one or more of the one or more memories and executed by one or more of the one or more processors that together collect and store metric data by
receiving and storing metric data points in memory,
extracting previously received metric data points from memory, compressing the extracted metric data points, and storing the compressed metric data points in memory, the compressed metric data points each comprising a fixed number n of bits, where n is less than or equal to 8; and
retrieving compressed metric data points from memory and storing the retrieved compressed metric data points in one or more of the one or more data-storage devices.

2. The metric-data collection-and-storage subsystem of claim 1 wherein each metric data point is representable as a timestamp/metric-data-value pair.

3. The metric-data collection-and-storage subsystem of claim 2 wherein the metric-data collection-and-storage subsystem provides, to a requesting data-analysis subsystem:
uncompressed metric data points, extracted from memory, corresponding to a first time window;
compressed data points, extracted from memory, corresponding to a second time window earlier in time than the first time window; and
compressed data points, extracted from mass storage, corresponding to a third time window earlier in time than the second time window.

4. The metric-data collection-and-storage subsystem of claim 2 wherein each metric data point is compressed to a single one-bit value that indicates whether or not the metric data point is a significant metric data point.

5. The metric-data collection-and-storage subsystem of claim 4 wherein a significant data point is one of:
a peak data point;
a trough data point; or
an inflection data point.

6. The metric-data collection-and-storage subsystem of claim 2 wherein a metric data point is compressed to a single two-bit value that indicates the relationship of the of the metric data point's metric-data value to the metric-data value of a preceding metric data point that precedes the metric data point in a time series of metric data points.

7. The metric-data collection-and-storage subsystem of claim 6 wherein, of the four possible numerical values of the two-bit value, a first numerical value represents an equality or an approximate equality relationship, a second numerical value represents a less-than relationship, and a third numerical value represents a greater-than relationship.

8. The metric-data collection-and-storage subsystem of claim 7 wherein a fourth numerical value represents a particular type of significant data point.

9. The metric-data collection-and-storage subsystem of claim 2
wherein the metric-data collection-and-storage subsystem stores uncompressed metric data points in a raw-metric-data circular queue; and
wherein the metric-data collection-and-storage subsystem stores compressed metric data points in a compressed-metric-data circular queue.

10. The metric-data collection-and-storage subsystem of claim 9 wherein multiple compressed metric data points are stored in each unit of data storage, such as a 32-bit word or a 64-bit word, within a the compressed-metric-data circular queue.

11. A method that efficiently stores metric data points in a computer system having one or more processors; one or more memories; and one or more data-storage devices, the method comprising:
receiving and storing the metric data points in memory,
extracting previously received metric data points from memory, compressing the extracted metric data points, and storing the compressed metric data points in memory, the compressed metric data points each comprising a fixed number n of bits, where n is less than or equal to 8; and
retrieving compressed metric data points from memory and storing the retrieved compressed metric data points in one or more of the one or more data-storage devices.

12. The method of claim 11 wherein each metric data point is representable as a timestamp/metric-data-value pair.

13. The method of claim 12 further comprising:
providing, to a requesting data-analysis subsystem, uncompressed metric data points corresponding to a first time window;
compressed data points, extracted from memory, corresponding to a second time window earlier in time than the first time window; and
compressed data points, extracted from mass storage, corresponding to a third time window earlier in time than the second time window.

14. The method of claim 12 further comprising compressing each metric data point is compressed to a single one-bit value that indicates whether or not the metric data point is a significant metric data point.

15. The method of claim 14 wherein a significant data point is one of:
a peak data point;
a trough data point; or
an inflection data point.

16. The method of claim 12 further comprising compressing each metric data point to a single two-bit value that indicates the relationship of the of the metric data point's metric-data value to the metric-data value of a preceding metric data point that precedes the metric data point in a time series of metric data points.

17. The method of claim 16 wherein, of the four possible numerical values of the two-bit value, a first numerical value represents an equality or an approximate equality relationship, a second numerical value represents a less-than relationship, and a third numerical value represents a greater-than relationship.

18. The method of claim 17 wherein a fourth numerical value represents a particular type of significant data point.

19. A physical data-storage device that stores a sequence of computer instructions that, when executed by one or more processors within one or more computer systems that each includes one or more processors, one or more memories, and one or more data-storage devices, control the one or more computer systems to:
receive and store the metric data points in memory,
extract previously received metric data points from memory, compress the extracted metric data points, and store the compressed metric data points in memory, the compressed metric data points each comprising a fixed number n of bits, where n is less than or equal to 8; and
retrieve compressed metric data points from memory and store the retrieved compressed metric data points in one or more of the one or more data-storage devices.

20. The physical data-storage device of claim 19 wherein the computer instructions further control the one or more computer systems to:
provide, to a requesting data-analysis subsystem, uncompressed metric data points corresponding to a first time window;
compress data points, extracted from memory, corresponding to a second time window earlier in time than the first time window; and
compress data points, extracted from mass storage, corresponding to a third time window earlier in time than the second time window,
receiving and storing metric data points in memory,
compressing previously received metric data points and storing the compressed metric data points in memory, the compressed metric data points each comprising a fixed number n of bits, where n is less than or equal to 8; and
retrieving compressed metric data points from memory and storin

Assignees

• Vmware Inc

Inventors

• Pedersen Paul
• Brown Darren
• Li Wei
• Nutman Leah
• Nakai Sergio

Classifications

• G06F3/0661 Primary

  Format or protocol conversion arrangements · CPC title

• G06F3/0683

  Plurality of storage devices · CPC title

• G06F3/0608 Primary

  Saving storage space on storage systems · CPC title

• G06F3/067

  Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS] · CPC title