Drift detection for predictive network models

Patent metadata

Field	Value
Publication number	US-11722359-B2
Application number	US-202117479297-A
Country	US
Kind code	B2
Filing date	Sep 20, 2021
Priority date	Sep 20, 2021
Publication date	Aug 8, 2023
Grant date	Aug 8, 2023

Abstract

A method, computer system, and computer program product are provided for detecting drift in predictive models for network devices and traffic. A plurality of streams of time-series telemetry data are obtained, the time-series telemetry data generated by network devices of a data network. The plurality of streams are analyzed to identify a subset of streams, wherein each stream of the subset of streams includes telemetry data that is substantially empirically distributed. The subset of streams of time-series data are analyzed to identify a change point. In response to identifying the change point, additional time-series data is obtained from one or more streams of the plurality of streams of time-series telemetry data. A predictive model is trained using the additional time-series data to update the predictive model and provide a trained predictive model.

First claim

What is claimed is:

1. A computer-implemented method comprising:
obtaining a plurality of streams of time-series telemetry data, the time-series telemetry data generated by network devices of a data network;
analyzing the plurality of streams to identify a subset of streams, wherein each stream of the subset of streams includes telemetry data that substantially matches an empirical distribution function;
analyzing the subset of streams of time-series data to identify a change point by:
computing a matrix profile using the subset of streams of time-series data, and
identifying a plurality of windows based on a repeating pattern of the subset of streams of time-series data;

in response to identifying the change point, obtaining additional time-series data from one or more streams of the plurality of streams of time-series telemetry data; and
re-training a predictive model using the additional time-series data to update the predictive model and provide a trained predictive model.

2. The computer-implemented method of claim 1 , wherein analyzing the subset of streams to identify the change point comprises comparing time-series data of one window to time-series data of a previous window to determine a density ratio on consecutive time intervals.

3. The computer-implemented method of claim 1 , further comprising:
applying the trained predictive model to identify network events of interest.

4. The computer-implemented method of claim 1 , wherein analyzing the subset of streams to identify the change point is further based on an occurrence of one or more predefined network events.

5. The computer-implemented method of claim 1 , wherein analyzing the subset of streams to identify the change point comprises comparing an occurrence of one or more network anomalies with respect to an expected repair time for the one or more network anomalies.

6. The computer-implemented method of claim 1 , further comprising:
in response to identifying the change point, deactivating a current predictive model and applying a fallback model to identify network events of interest.

7. An apparatus comprising:
one or more computer processors;
a network interface configured to enable network communications;
one or more computer readable storage media; and
program instructions stored on the one or more computer readable storage media for execution by at least one of the one or more computer processors, the program instructions comprising instructions to:
obtain a plurality of streams of time-series telemetry data, the time-series telemetry data generated by network devices of a data network;
analyze the plurality of streams to identify a subset of streams, wherein each stream of the subset of streams includes telemetry data that substantially matches an empirical distribution function;
analyze the subset of streams of time-series data to identify a change point by:
computing a matrix profile using the subset of streams of time-series data, and
identifying a plurality of windows based on a repeating pattern of the subset of streams of time-series data;

in response to identifying the change point, obtain additional time-series data from one or more streams of the plurality of streams of time-series telemetry data; and
re-train a predictive model using the additional time-series data to update the predictive model and provide a trained predictive model.

8. The apparatus of claim 7 , wherein analyzing the subset of streams to identify the change point comprises comparing time-series data of one window to time-series data of a previous window to determine a density ratio on consecutive time intervals.

9. The apparatus of claim 7 , wherein the program instructions further comprise instructions to:
apply the trained predictive model to identify network events of interest.

10. The apparatus of claim 7 , wherein analyzing the subset of streams to identify the change point is further based on an occurrence of one or more predefined network events.

11. The apparatus of claim 7 , wherein analyzing the subset of streams to identify the change point comprises comparing an occurrence of one or more network anomalies with respect to an expected repair time for the one or more network anomalies.

12. The apparatus of claim 7 , wherein the program instructions further comprise instructions to:
in response to identifying the change point, deactivate a current predictive model and apply a fallback model to identify network events of interest.

13. One or more non-transitory computer readable storage media collectively having program instructions embodied therewith, the program instructions executable by a computer to cause the computer to:
obtain a plurality of streams of time-series telemetry data, the time-series telemetry data generated by network devices of a data network;
analyze the plurality of streams to identify a subset of streams, wherein each stream of the subset of streams includes telemetry data that substantially matches an empirical distribution function;
analyze the subset of streams of time-series data to identify a change point by:
computing a matrix profile using the subset of streams of time-series data, and
identifying a plurality of windows based on a repeating pattern of the subset of streams of time-series data;

in response to identifying the change point, obtain additional time-series data from one or more streams of the plurality of streams of time-series telemetry data; and
train a predictive model using the additional time-series data to update the predictive model and provide a trained predictive model.

14. The one or more non-transitory computer readable storage media of claim 13 , wherein analyzing the subset of streams to identify the change point comprises comparing time-series data of one window to time-series data of a previous window to determine a density ratio on consecutive time intervals.

15. The one or more non-transitory computer readable storage media of claim 13 , wherein the program instructions further cause the computer to:
apply the trained predictive model to identify network events of interest.

16. The one or more non-transitory computer readable storage media of claim 13 , wherein analyzing the subset of streams to identify the change point is further based on an occurrence of one or more predefined network events.

17. The one or more non-transitory computer readable storage media of claim 13 , wherein analyzing the subset of streams to identify the change point comprises comparing an occurrence of one or more network anomalies with respect to an expected repair time for the one or more network anomalies.

18. The computer-implemented method of claim 1 , wherein the additional time-series data that is used to re-train the predictive model is collected for a threshold amount of time after the change point is identified.

19. The apparatus of claim 7 , wherein the additional time-series data that is used to re-train the predictive model is collected for a threshold amount of time after the change point is identified.

20. The one or more non-transitory computer readable storage media of claim 13 , wherein the additional time-series data that is used to re-train the predictive model is collected for a threshold amount of time after the change point is identified.

Assignees

• Cisco Tech Inc

Inventors

• Fenoglio Enzo
• Zacks David John
• Gao Zizhen
• Pignataro Carlos M
• Goloubev Dmitry

Classifications

• H04L41/064 Primary

  involving time analysis · CPC title

• G06F18/214

  Generating training patterns; Bootstrap methods, e.g. bagging or boosting · CPC title

• H04L41/16 Primary

  using machine learning or artificial intelligence · CPC title

• H04L43/04

  Processing captured monitoring data, e.g. for logfile generation · CPC title

• H04L41/147

  for predicting network behaviour · CPC title