Continuous anomaly detection service

Patent metadata

Field	Value
Publication number	US-10200262-B1
Application number	US-201615206123-A
Country	US
Kind code	B1
Filing date	Jul 8, 2016
Priority date	Jul 8, 2016
Publication date	Feb 5, 2019
Grant date	Feb 5, 2019

Abstract

An anomaly detection system includes a plurality of signals. Each of the signals is associated with an anomaly detection procedure that will be used to identify anomalies within the signal. Anomaly detection is performed by applying the anomaly detection procedure to a sequential set of data points of a signal. The signals are updated based on incoming data streams. The data streams are analyzed, and the sequential set of data points for each signal is updated based on data points extracted from the data streams.

First claim

What is claimed is:

1. A computer-implemented method for performing anomaly detection, comprising:
receiving a plurality of data streams;
storing, for each of the data streams, one or more events in a field searchable data store;
determining one or more of the data streams associated with one or more corresponding signals of a plurality of signals;
identifying, from the one or more determined data streams, relevant events of the one or more events that are associated with the one or more corresponding signals;
identifying, for each of the one or more determined data streams, a corresponding set of data points by deriving values for the data points from machine data of the relevant events for the determined data stream;
inserting the identified sets of data points into the one or more corresponding signals; and
continuously performing anomaly detection on the plurality of signals.

2. The computer-implemented method of claim 1 , wherein each of the plurality of signals comprises a sequential set of data points, the method further comprising:
removing, for each of the one or more corresponding signals, a least recently received subset of data points of the sequential set of data points from the signal; and
updating, for each of the one or more corresponding signals, the sequential set of data points of the signal by inserting the identified set of data points into a beginning position of the sequential set of data points, and wherein continuously performing anomaly detection comprises continuously performing anomaly detection for each of the one or more corresponding signals based on the updated sequential sets of data points.

3. The computer-implemented method of claim 2 , wherein, for each of the one or more corresponding signals, the number of removed data points equals the number of data points in the identified set of data points.

4. The computer-implemented method of claim 3 , wherein, for each memory location associated with one of the removed data points, the removed data point is replaced with one of the updated data points.

5. The computer-implemented method of claim 2 , wherein updating the sequential set of data points with the updated data points comprises:
determining a sequential ordering of the updated data points; and
inserting the updated data points into the sequential set of data points based on the determined sequential ordering.

6. The computer-implemented method of claim 2 , wherein each sequential set of data points is stored in a first in first out (FIFO) queue.

7. The computer-implemented method of claim 6 , wherein each FIFO queue comprises a circular queue.

8. The computer-implemented method of claim 6 , wherein, for each sequential set of data points, a pointer defines a starting point for the FIFO queue.

9. The computer-implemented method of claim 1 , wherein each of the plurality of signals comprises a sequential set of data points, and wherein each signal has a required number of data points, further comprising:
determining, for each signal, whether the sequential set of data points has less than the required number of data points; and
appending, for each of the plurality of signals that has less than the required number of data points, one or more historical data points to the sequential set of data points.

10. The computer-implemented method of claim 9 , wherein appending, for each of the plurality of signals that has less than the required number of data points, the one or more historical data points to the sequential set of data points comprises:
determining a number of additional data points that are required for the sequential set of data points to equal the required number of data points;
determining a sequential ordering for the one or more historical data points within the sequential set of data points;
accessing the one or more historical data points based on the number of additional data points and the sequential ordering; and
appending the one or more historical data points to the sequential set of data points based on the sequential ordering.

11. The computer-implemented method of claim 10 , wherein accessing the one or more historical data points based on the number of additional data points and the sequential ordering comprises:
requesting the historical data points from the field searchable data store based on the number of additional data points and the sequential ordering; and
receiving the historical data points from the field searchable data store.

12. The computer-implemented method of claim 1 , wherein performing anomaly detection comprises performing a cohesive anomaly detection procedure to determine an anomaly result by:
identifying a signal group, wherein the signal group comprises signals of a subset of the plurality of signals that are associated with the cohesive anomaly detection procedure; and
determining the anomaly result based on a comparison of the signals of the signal group.

13. The computer-implemented method of claim 12 , wherein identifying an anomaly based on a comparison of the signals of the signal group comprises:
identifying, for each of the signals of the signal group, one or more comparison data points; and
determining the anomaly result based on a comparison of the comparison data points of the signals of the signal group.

14. The computer-implemented method of claim 1 ,
wherein performing anomaly detection comprises performing a trending anomaly detection procedure to determine an anomaly result by:
determining an anomaly detection score based on a changes in data points of the one or more corresponding signals over time; and
determining the anomaly result based on a comparison of the changes to one or more thresholds.

15. The computer-implemented method of claim 1 , wherein determining one or more of the data streams associated with one or more corresponding signals comprises searching each data stream based on corresponding key performance indicators for the one or more corresponding signals.

16. The computer-implemented method of claim 1 , further comprising accessing a plurality of anomaly detection configurations, wherein each of the anomaly detection configurations corresponds to one of the plurality of signals, and wherein each anomaly detection configuration determines a corresponding anomaly detection procedure and a plurality of anomaly detection parameters associated with the plurality of signals.

17. The computer-implemented method of claim 16 , wherein the anomaly detection parameters comprise an anomaly detection threshold, a signal length, or an alert setting.

18. The computer-implemented method of claim 1 , wherein the corresponding set of data points is identified by:
generating a search command based on an anomaly detection configuration for the one or more corresponding signals; and
periodically executing the search command to extract the set of data points.

19. A non-transitory computer-readable storage medium comprising instructions stored thereon, which when executed by one or more processors, cause the one or more processors to perform operations comprising:
receiving a plurality of data streams;
storing, for each of the data streams, one or more events in a field searchable data store;
determining one or more, for each of the data streams associated with one or more corresponding signals of a plurality of signals;
identifying, from the one or more determined data streams, relevant events of the one or more events that are associated with the one or more corresponding signals;
identifying, fo

Assignees

• Splunk Inc
• Splunk Inc

Inventors

• Leverich Jacob Barton
• Cai shang
• Zhang Hongyang
• Ganea Mihai
• Cruise Alex

Classifications

• H04L43/106

  using time related information in packets, e.g. by adding timestamps · CPC title

• H04L41/0686

  Additional information in the notification, e.g. enhancement of specific meta-data · CPC title

• H04L43/16

  Threshold monitoring · CPC title

• H04L43/0823 Primary

  Errors, e.g. transmission errors · CPC title

• H04L67/2804

  Electricity · mapped topic