Assessing data fidelity in a machine learning-based network assurance system

Patent metadata

Field	Value
Publication number	US-11397876-B2
Application number	US-201916692165-A
Country	US
Kind code	B2
Filing date	Nov 22, 2019
Priority date	Nov 22, 2019
Publication date	Jul 26, 2022
Grant date	Jul 26, 2022

Abstract

In one embodiment, a service computes a data fidelity metric for network telemetry data used by a machine learning model to monitor a computer network. The service detects unacceptable performance of the machine learning model. The service determines a correlation between the data fidelity metric and the unacceptable performance of the machine learning model. The service adjusts generation of the network telemetry data for input to the machine learning model, based on the determined correlation between the data fidelity metric and the unacceptable performance of the machine learning model.

First claim

What is claimed is:

1. A method comprising:
computing, by a service, a data fidelity metric for network telemetry data used by a machine learning model to monitor a computer network, wherein the data fidelity metric is computed based in part on missing values in the network telemetry data;
detecting, by the service, unacceptable performance of the machine learning model;
determining, by the service, a correlation between the data fidelity metric and the unacceptable performance of the machine learning model; and
adjusting, by the service, generation of the network telemetry data for input to the machine learning model, based on the determined correlation between the data fidelity metric and the unacceptable performance of the machine learning model.

2. The method as in claim 1 , wherein the machine learning model comprises an anomaly detector, and wherein the unacceptable performance of the machine learning model corresponds to the anomaly detector detecting more than a threshold percentage of anomalies.

3. The method as in claim 1 , wherein determining a correlation between the data fidelity metric and the unacceptable performance of the machine learning model comprises:
predicting whether performance of the machine learning model will be unacceptable, given a particular input set of network telemetry data.

4. The method as in claim 1 , wherein the data fidelity metric is computed based in part on a sampling frequency at which the network telemetry data is collected in the network, and wherein adjusting generation of the network telemetry data comprises:
adjusting the sampling frequency at which the network telemetry data is collected in the network.

5. The method as in claim 1 , wherein adjusting generation of the network telemetry data comprises:
using a regression model to predict the missing values, prior to input to the machine learning model.

6. The method as in claim 1 , wherein adjusting generation of the network telemetry data comprises:
sending an instruction to an entity in the network to collect the missing values.

7. The method as in claim 1 , wherein the data fidelity metric is computed based in part on spurious data in the network telemetry data, and wherein adjusting generation of the network telemetry data comprises:
using a regression model to reconstruct the spurious data, prior to input to the machine learning model.

8. The method as in claim 1 , wherein the machine learning model is part of a network assurance system.

9. The method as in claim 1 , wherein the machine learning model is configured to classify devices in the network by device type.

10. An apparatus, comprising:
one or more network interfaces;
a processor coupled to the network interfaces and configured to execute one or more processes; and
a memory configured to store a process executable by the processor, the process when executed configured to:
compute a data fidelity metric for network telemetry data used by a machine learning model to monitor a computer network, wherein the data fidelity metric is computed based in part on missing values in the network telemetry data;
detect unacceptable performance of the machine learning model;
determine a correlation between the data fidelity metric and the unacceptable performance of the machine learning model; and
adjust generation of the network telemetry data for input to the machine learning model, based on the determined correlation between the data fidelity metric and the unacceptable performance of the machine learning model.

11. The apparatus as in claim 10 , wherein the machine learning model comprises an anomaly detector, and wherein the unacceptable performance of the machine learning model corresponds to the anomaly detector detecting more than a threshold percentage of anomalies.

12. The apparatus as in claim 10 , wherein the apparatus determines a correlation between the data fidelity metric and the unacceptable performance of the machine learning model by:
predicting whether performance of the machine learning model will be unacceptable, given a particular input set of network telemetry data.

13. The apparatus as in claim 10 , wherein the data fidelity metric is computed based in part on a sampling frequency at which the network telemetry data is collected in the network, and wherein the apparatus adjusts generation of the network telemetry data by:
adjusting the sampling frequency at which the network telemetry data is collected in the network.

14. The apparatus as in claim 10 , wherein the apparatus adjusts generation of the network telemetry data by:
using a regression model to predict the missing values, prior to input to the machine learning model.

15. The apparatus as in claim 10 , wherein the apparatus adjusts generation of the network telemetry data by:
sending an instruction to an entity in the network to collect the missing values.

16. The apparatus as in claim 10 , wherein the data fidelity metric is computed based in part on spurious data in the network telemetry data, and wherein the apparatus adjusts generation of the network telemetry data by:
using a regression model to reconstruct the spurious data, prior to input to the machine learning model.

17. The apparatus as in claim 10 , wherein the machine learning model is part of a network assurance system or is configured to classify devices in the network with device types.

18. A tangible, non-transitory, computer-readable medium storing program instructions that cause a service to execute a process comprising:
computing, by a service, a data fidelity metric for network telemetry data used by a machine learning model to monitor a computer network, wherein the data fidelity metric is computed based in part on missing values in the network telemetry data;
detecting, by the service, unacceptable performance of the machine learning model;
determining, by the service, a correlation between the data fidelity metric and the unacceptable performance of the machine learning model; and
adjusting, by the service, generation of the network telemetry data for input to the machine learning model, based on the determined correlation between the data fidelity metric and the unacceptable performance of the machine learning model.

Assignees

• Cisco Tech Inc

Inventors

• Vasseur Jean-Philippe
• Kolar Vinay Kumar
• Di Pietro Andrea
• Mermoud Grégory
• Savalle Pierre-Andre

Classifications

• H04L41/16 Primary

  using machine learning or artificial intelligence · CPC title

• G06F18/2193

  based on specific statistical tests · CPC title

• G06F18/217

  Validation; Performance evaluation; Active pattern learning techniques · CPC title

• G06N3/044

  Recurrent networks, e.g. Hopfield networks · CPC title

• G06N3/09

  Supervised learning · CPC title