Data-agnostic adjustment of hard thresholds based on user feedback

Patent metadata

Field	Value
Publication number	US-10467119-B2
Application number	US-201715479182-A
Country	US
Kind code	B2
Filing date	Apr 4, 2017
Priority date	Jun 24, 2014
Publication date	Nov 5, 2019
Grant date	Nov 5, 2019

Abstract

This disclosure is directed to data-agnostic computational methods and systems for adjusting hard thresholds based on user feedback. Hard thresholds are used to monitor time-series data generated by a data-generating entity. The time-series data may be metric data that represents usage of the data-generating entity over time. The data is compared with a hard threshold associated with usage of the resource or process and when the data violates the threshold, an alert is typically generated and presented to a user. Methods and systems collect user feedback after a number of alerts to determine the quality and significance of the alerts. Based on the user feedback, methods and systems automatically adjust the hard thresholds to better represent how the user perceives the alerts.

First claim

The invention claimed is:

1. A method stored in one or more data-storage devices and executed using one or more processors of a computing environment to adjust hard thresholds based on user feedback, the method comprising:
generating alerts when time-series data generated by a data-generating entity violates a hard threshold;
collecting user feedback from a survey presented to a user of the data-generating entity following each of the alerts;
when a number of user feedbacks generated by the user is greater than an average number of feedbacks per user of the data-generating entity, determining an adjusted hard threshold based on the user feedback; and
using the adjusted hard threshold to generate an alert indicating abnormal behavior of the data-generating entity, when time-series data generated by the data-generating entity violates the adjusted hard threshold.

2. The method of claim 1 , wherein generating alerts when the time-series data violates the hard threshold comprises one of:
when the hard threshold is an upper hard threshold, generating an alert when a portion of the data is greater than the upper hard threshold; and
when the hard threshold is a lower hard threshold, generating an alert when a portion of the data is less than the lower hard threshold.

3. The method of claim 1 , wherein collecting user feedback comprises presenting the user with survey questions regarding indicativeness, criticality, timeliness, and duration of each alert.

4. The method of claim 1 , wherein the determining the adjusted hard threshold based on the user feedback comprises:
calculating an alert confidence value based on the user feedback;
calculating the adjusted hard threshold from the hard threshold and a step size value greater than zero when the alert confidence is greater than zero; and
setting the hard threshold equal to the adjusted hard threshold.

5. The method of claim 4 , wherein calculating the alert confidence value comprises:
determining feedback statistics from values assigned to user feedback regarding indicativeness, criticality, timeliness, and duration of each alert;
calculating weighted statistics from the feedback statistics;
forming a histogram of the weighted statistics distributed over a number of subintervals;
calculating normalized frequencies of the weighted statistics based on the distribution of the weighted statistics;
calculating an entropy value of the weighted statistics; and
calculating a confidence value based on the entropy value of the weighted statistics.

6. The method of claim 5 wherein determining feedback statistics comprises:
generating sets of user feedback statistics regarding criticality, timeliness, and duration of the number of alerts based on the user feedback;
calculating a criticality confidence, timeliness confidence, and duration confidence based on corresponding feedback statistics;
calculating adjusted criticality when the criticality confidence is greater than zero zero;
calculating adjusted timeliness when the timeliness confidence is greater than zero; and
calculating adjusted duration when the duration confidence is greater than zero.

7. The method of claim 4 , wherein calculating the adjusted hard threshold comprises
calculating an average of weighted statistics based on the feedback statistics when the alert confidence is greater than zero;
calculating a noise degree from the average of the weighted statistics;
when the hard threshold is an upper hard threshold,
decreasing the hard threshold, when a difference between the noise degree and a user-defined noise tolerance is negative valued and outside a tolerance interval;
increasing the hard threshold, when the difference between the noise degree and the user-defined noise tolerance is positive valued and outside the tolerance interval; and

calculating the adjusted hard threshold as a function of the average of the weighted statistics, the alert confidence, and one of the increased and decreased hard threshold.

8. The method of claim 4 , wherein calculating the adjusted hard threshold comprises:
calculating an average of weighted statistics based on the feedback statistics when the alert confidence is greater than zero;
calculating a noise degree from the average of the weighted statistics;
when the hard threshold is a lower hard threshold,
increasing the hard threshold, when a difference between the noise degree and a user-defined noise tolerance is negative valued and outside a tolerance interval;
decreasing the hard threshold, when the difference between the noise degree and the user-defined noise tolerance is positive valued and outside the tolerance interval; and

calculating the adjusted hard threshold as a function of the average of the weighted statistics, the alert confidence, and one of the increased and decreased hard threshold.

9. A system that adjusts a hard threshold of a data-generating entity, the system comprising:
one or more processors;
one or more data-storage devices; and
a routine stored in the data-storage devices that when executed using the one or more processors perform operations comprising:
generating alerts when time-series data generated by a data-generating entity violates a hard threshold;
collecting user feedback from a survey presented to a user of the data-generating entity following each of the alerts;
when a number of user feedbacks generated by the user is greater than an average number of feedbacks per user of the data-generating entity, determining an adjusted hard threshold based on the user feedback; and
using the adjusted hard threshold to generate an alert indicating abnormal behavior of the data-generating entity, when time-series data generated by the data-generating entity violates the adjusted hard threshold.

10. The system of claim 9 , wherein generating alerts when the time-series data violates the hard threshold comprises one of:
when the hard threshold is an upper hard threshold, generating an alert when a portion of the data is greater than the upper hard threshold; and
when the hard threshold is a lower hard threshold, generating an alert when a portion of the data is less than the lower hard threshold.

11. The system of claim 9 , wherein collecting user feedback comprises presenting the user with questions regarding indicativeness, criticality, timeliness, and duration of each alert.

12. The system of claim 9 , wherein determining the adjusted hard threshold based on the user feedback comprises:
calculating an alert confidence value based on the user feedback;
calculating the adjusted hard threshold from the hard threshold and a step size value greater than zero when the alert confidence is greater than zero; and
setting the hard threshold equal to the adjusted hard threshold.

13. The system of claim 12 , wherein calculating the alert confidence value comprises:
determining feedback statistics from values assigned to user feedback regarding indicativeness, criticality, timeliness, and duration of each alert;
calculating weighted statistics from the feedback statistics;
forming a histogram of the weighted statistics distributed over a number of subintervals;
calculating normalized frequencies of the weighted statistics based on the distribution of the weighted statistics;
calculating an entropy value of the weighted statistics; and
calculating a confidence value based on the entropy value of the weighted statistics.

14. The system of claim 13 wherein determining feedback statistics comprises:
generating sets of user feedback statistics regarding criticality, timeliness, and duration of the number of alerts

Assignees

• Vmware Inc

Inventors

• Grigoryan Naira Movses
• Marvasti Mazda A
• Poghosyan Arnak
• Harutyunyan Ashot Nshan
• Yankov Yanislav

Classifications

• G06F11/3438

  monitoring of user actions (tracking the activity of the user H04L67/535) · CPC title

• G06F11/07

  Responding to the occurrence of a fault, e.g. fault tolerance · CPC title

• G06F11/324

  Display of status information · CPC title

• G06F2201/81

  Threshold · CPC title

• G06F11/3452 Primary

  Performance evaluation by statistical analysis · CPC title