Adaptive alert monitoring

Patent metadata

Field	Value
Publication number	US-10979281-B2
Application number	US-201916264369-A
Country	US
Kind code	B2
Filing date	Jan 31, 2019
Priority date	Jan 31, 2019
Publication date	Apr 13, 2021
Grant date	Apr 13, 2021

Abstract

Various embodiments provide for alert generation based on alert dependency. For some embodiments, the alert dependency checking facilitates alert noise reduction. Various embodiments described herein dynamically find or discover alert dependencies based on one or more alerts currently active, one or more active alerts generated in the past, or some combination of both. Various embodiments described herein provide alert monitoring that adapts based on an alert state of a machine. Various embodiments described herein generate a health score for a machine based on an alert state of the machine. Various embodiments described herein provide a tool for managing definitions of one or more alerts that can be identified as an active alert for a machine.

First claim

What is claimed is:

1. A method comprising:
periodically monitoring, by one or more hardware processors, a computing node for a set of active alerts based on a monitoring frequency, the computing node operating one or more software services, each alert of the set of active alerts comprising a notification generated by the computing node based on a violation of an invariant of the computing node;
generating, by the one or more hardware processors, an alert state for the computing node based on the set of active alerts, the alert state describing the set of active alerts with respect to a set of identifiable alerts for the computing node, the alert state comprising a vector representing a plurality of statuses corresponding to the set of identifiable alerts for the computing node;
adjusting, by the one or more hardware processors, the monitoring frequency based on the alert state; and
generating, by the one or more hardware processors, a health score for the computing node based on the alert state, the generating the health score comprising:
applying to the alert state a set of weights that corresponds to the set of identified alerts to generate intermediate alert state data; and
calculating a dot product of the alert state based on the intermediate alert state data to generate the health score.

2. The method of claim 1 , wherein the adjusting the monitoring frequency based on the alert state comprises:
determining whether the alert state represents a change in comparison to a prior determined alert state; and
adjusting the monitoring frequency based on the determining of whether the alert state represents the change.

3. The method of claim 2 , wherein the adjusting the monitoring frequency based on the determining of whether the alert state represents the change comprises:
increasing the monitoring frequency in response to the change indicating an increase in active alerts.

4. The method of claim 2 , wherein the adjusting the monitoring frequency based on the determining of whether the alert state represents the change comprises:
decreasing the monitoring frequency in response to the change indicating a decrease in active alerts.

5. The method of claim 1 , wherein each status of the plurality of statuses comprises a value of on or off.

6. The method of claim 1 , wherein the periodically monitoring the computing node for the set of active alerts based on the monitoring frequency comprises:
accessing telemetry data associated with the computing node, the telemetry data comprising a log data generated by the computing node; and
evaluating the telemetry data to identify a set of triggered alerts triggered by the computing node;
accessing alert dependency data that describes a set of dependencies between a plurality of alerts; and
filtering the set of triggered alerts based on the alert dependency data to generate the set of active alerts.

7. The method of claim 6 , wherein the evaluating the telemetry data to identify the set of triggered alerts for the computing node comprises:
generating time-series alert data based on the telemetry data to identify the set of triggered alerts, the generating the time-series alert data based on the telemetry data to identify the set of triggered alerts comprising:
for a given identifiable alert in the set of identifiable alerts, recording a time for each time a given metric associated the given identifiable alert surpasses a threshold value.

8. The method of claim 7 , further comprising:
adjusting, by the one or more hardware processors, the threshold value based on the alert state.

9. The method of claim 7 , further comprising:
adjusting, by the one or more hardware processors, the threshold value based on a temporary override selected by a user, the temporary override temporarily changing the threshold value.

10. A system comprising:
a memory storing instructions; and
one or more hardware processors communicatively coupled to the memory and configured by the instructions to perform operations comprising:
periodically monitoring a computing node for a set of active alerts based on a monitoring frequency, the computing node operating one or more software services, each alert of the set of active alerts comprising a notification generated by the computing node based on a violation of an invariant of the computing node;
generating an alert state for the computing node based on the set of active alerts, the alert state describing the set of active alerts with respect to a set of identifiable alerts for the computing node, the alert state comprising a vector representing a plurality of statuses corresponding to the set of identifiable alerts for the computing node;
adjusting the monitoring frequency based on the alert state; and
generating a health score for the computing node based on the alert state, the generating the health score comprising:
applying to the alert state a set of weights that corresponds to the set of identified alerts to generate intermediate alert state data; and

calculating a dot product of the alert state based on the intermediate alert state data to generate the health score.

11. The system of claim 10 , wherein the adjusting the monitoring frequency based on the alert state comprises:
determining whether the alert state represents a change in comparison to a prior determined alert state; and
adjusting the monitoring frequency based on the determining of whether the alert state represents the change.

12. The system of claim 10 , wherein the periodically monitoring the computing node for the set of active alerts based on the monitoring frequency comprises:
accessing telemetry data associated with the computing node, the telemetry data comprising a log data generated by the computing node; and
evaluating the telemetry data to identify a set of triggered alerts triggered by the computing node;
accessing alert dependency data that describes a set of dependencies between a plurality of alerts; and
filtering the set of triggered alerts based on the alert dependency data to generate the set of active alerts.

13. The system of claim 12 , wherein the evaluating the telemetry data to identify the set of triggered alerts for the computing node comprises:
generating time-series alert data based on the telemetry data to identify the set of triggered alerts, the generating the time-series alert data based on the telemetry data to identify the set of triggered alerts comprising:
for a given identifiable alert in the set of identifiable alerts, recording a time for each time a given metric associated the given identifiable alert surpasses a threshold value.

14. The system of claim 13 , wherein the operations further comprises: adjusting the threshold value based on the alert state.

15. The system of claim 13 , wherein the operations further comprises: adjusting the threshold value based on a temporary override.

16. A non-transitory computer-readable storage medium comprising instructions that, when executed by a processing device, cause the processing device to perform operations comprising:
periodically monitoring a computing node for a set of active alerts based on a monitoring frequency, the computing node operating one or more software services, each alert of the set of active alerts comprising a notification generated by the computing node based on a violation of an invariant of the computing node;
generating an alert state for the computing node based on the set of active alerts, the alert state describing the set of active alerts with respect to a set of identifiable alerts for the computing node, the

Assignees

• Rubrik Inc

Inventors

• Arora Gurjeet S
• Bavishi Karan Jayesh
• Talamas Cano Daniel
• Louie John
• Joshi Chetas
• Noe Matthew Edward

Classifications

• H04L43/20

  the monitoring system or the monitored elements being virtualised, abstracted or software-defined entities, e.g. SDN or NFV · CPC title

• H04L41/40

  using virtualisation of network functions or resources, e.g. SDN or NFV entities · CPC title

• H04L43/028

  by filtering · CPC title

• H04L43/0817

  by checking functioning · CPC title

• H04L41/0604 Primary

  using filtering, e.g. reduction of information by using priority, element types, position or time · CPC title