Data-agnostic adjustment of hard thresholds based on user feedback

Patent metadata

Field	Value
Publication number	US-9632905-B2
Application number	US-201414312815-A
Country	US
Kind code	B2
Filing date	Jun 24, 2014
Priority date	Jun 24, 2014
Publication date	Apr 25, 2017
Grant date	Apr 25, 2017

Abstract

This disclosure is directed to data-agnostic computational methods and systems for adjusting hard thresholds based on user feedback. Hard thresholds are used to monitor time-series data generated by a data-generating entity. The time-series data may be metric data that represents usage of the data-generating entity over time. The data is compared with a hard threshold associated with usage of the resource or process and when the data violates the threshold, an alert is typically generated and presented to a user. Methods and systems collect user feedback after a number of alerts to determine the quality and significance of the alerts. Based on the user feedback, methods and systems automatically adjust the hard thresholds to better represent how the user perceives the alerts.

First claim

The invention claimed is:

1. A method stored in one or more data-storage devices and executed using one or more processors of a computing environment, the method comprising:
generating alerts when a portion of time-series data generated by a data-generating entity is greater than an upper hard threshold or less than a lower hard threshold;
collecting user feedback for a number of the alerts;
generating a set of user feedback statistics based on the user feedback;
calculating an alert confidence based on the feedback statistics; and
calculating an adjusted hard threshold based on the hard threshold when the alert confidence is greater than zero.

2. The method of claim 1 , wherein collecting user feedback further comprises one or presenting a user with one or more survey questions for each of the number of alerts and monitor the user's activities following each of the number of alerts.

3. The method of claim 1 , wherein generating the set of user feedback statistics further comprises assigning a numerical value to each answer a user gives to one or more survey questions regarding an alert, each numerical value is a user feedback statistic in the set of feedback statistics.

4. The method of claim 1 , wherein calculating the alert confidence further comprises
calculating weighted statistics for the set of feedback statistics;
forming a histogram of the weighted statistics distributed over a number of subintervals;
calculating normalized frequencies of the weighted statistics based on the distribution of the weighted statistics;
calculating an entropy value of the weighted statistics; and
calculating a confidence value based on entropy value of the weighted statistics.

5. The method of claim 1 , wherein calculating the adjusted hard threshold further comprises
calculating an average of weighted statistics based on the feedback statistics when the alert confidence is greater than zero;
calculating a noise degree from the average of the weighted statistics;
when the hard threshold is an upper hard threshold,
decreasing the hard threshold, when a difference between the noise degree and a user-defined noise tolerance is negative valued and outside a tolerance interval;
increasing the hard threshold, when the difference between the noise degree and the user-defined noise tolerance is positive valued and outside the tolerance interval; and

calculating the adjusted hard threshold as a function of the average of the weighted statistics, the alert confidence, and one of the increased and decreased hard threshold.

6. The method of claim 1 , wherein calculating the adjusted hard threshold further comprises
calculating an average of weighted statistics based on the feedback statistics when the alert confidence is greater than zero;
calculating a noise degree from the average of the weighted statistics;
when the hard threshold is a lower hard threshold,
increasing the hard threshold, when a difference between the noise degree and a user-defined noise tolerance is negative valued and outside a tolerance interval;
decreasing the hard threshold, when the difference between the noise degree and the user-defined noise tolerance is positive valued and outside the tolerance interval; and

calculating the adjusted hard threshold as a function of the average of the weighted statistics, the alert confidence, and one of the increased and decreased hard threshold.

7. The method of claim 1 further comprises
generating sets of user feedback statistics regarding criticality, timeliness, and duration of the number of alerts based on the user feedback;
calculating a criticality confidence, timeliness confidence, and duration confidence based on corresponding feedback statistics;
calculating adjusted criticality when the criticality confidence is greater than zero
calculating adjusted timeliness when the timeliness confidence is greater than zero; and
calculating adjusted duration when the duration confidence is greater than zero.

8. A system for adjusting a hard threshold comprising:
one or more processors;
one or more data-storage devices; and
a routine stored in the one or more data-storage devices and that when executed using the one or more processors, the routine controls the system to carry out
generating alerts when a portion of time-series data generated by a data-generating entity is greater than an upper hard threshold or less than a lower hard threshold;
collecting user feedback for a number of the alerts;
generating a set of user feedback statistics based on the user feedback;
calculating an alert confidence based on the feedback statistics; and
calculating an adjusted hard threshold based on the hard threshold when the alert confidence is greater than zero.

9. The system of claim 8 , wherein collecting user feedback further comprises one or presenting a user with one or more survey questions for each of the number of alerts and monitor the user's activities following each of the number of alerts.

10. The system of claim 8 , wherein generating the set of user feedback statistics further comprises assigning a numerical value to each answer a user gives to one or more survey questions regarding an alert, each numerical value is a user feedback statistic in the set of feedback statistics.

11. The system of claim 8 , wherein calculating the alert confidence further comprises
calculating weighted statistics for the set of feedback statistics;
forming a histogram of the weighted statistics distributed over a number of subintervals;
calculating normalized frequencies of the weighted statistics based on the distribution of the weighted statistics;
calculating an entropy value of the weighted statistics; and
calculating a confidence value based on entropy value of the weighted statistics.

12. The system of claim 8 , wherein calculating the adjusted hard threshold further comprises
calculating an average of weighted statistics based on the feedback statistics when the alert confidence is greater than zero;
calculating a noise degree from the average of the weighted statistics;
when the hard threshold is an upper hard threshold,
decreasing the hard threshold, when a difference between the noise degree and a user-defined noise tolerance is negative valued and outside a tolerance interval, decreasing the hard threshold;
increasing the hard threshold, when the difference between the noise degree and the user-defined noise tolerance is positive valued and outside the tolerance interval, increasing the hard threshold; and

calculating the adjusted hard threshold as a function of the average of the weighted statistics, the alert confidence, and one of the increased and decreased hard threshold.

13. The system of claim 8 , wherein calculating the adjusted hard threshold further comprises
calculating an average of weighted statistics based on the feedback statistics when the alert confidence is greater than zero;
calculating a noise degree from the average of the weighted statistics;
when the hard threshold is a lower hard threshold,
increasing the hard threshold, when a difference between the noise degree and a user-defined noise tolerance is negative valued and outside a tolerance interval, decreasing the hard threshold;
decreasing the hard threshold, when the difference between the noise degree and the user-defined noise tolerance is positive valued and outside the tolerance interval, increasing the hard threshold; and

calculating the adjusted hard threshold as a function of the average of the weighted statistics, the alert confidence, and one of the increased and decreased hard threshold.

14

Assignees

• Vmware Inc

Inventors

• Grigoryan Naira Movses
• Marvasti Mazda A
• Poghosyan Arnak
• Harutyunyan Ashot Nshan
• Yankov Yanislav

Classifications

• G06F2201/875

  Monitoring of systems including the internet · CPC title

• G06F11/3438

  monitoring of user actions (tracking the activity of the user H04L67/535) · CPC title

• G06F2201/81

  Threshold · CPC title

• G06F11/3433

  for load management (allocation of a server based on load conditions G06F9/505; load rebalancing G06F9/5083; redistributing the load in a network by a load balancer H04L67/1029) · CPC title

• G06F11/07

  Responding to the occurrence of a fault, e.g. fault tolerance · CPC title