Model training for on-premise execution in a network assurance system

Patent metadata

Field	Value
Publication number	US-11070441-B2
Application number	US-201916578565-A
Country	US
Kind code	B2
Filing date	Sep 23, 2019
Priority date	Sep 23, 2019
Publication date	Jul 20, 2021
Grant date	Jul 20, 2021

Abstract

In one embodiment, a network assurance service maintains a data lake of network telemetry data obtained by the service from any number of computer networks. The service generates a machine learning model for on-premise execution in a particular computer network to detect network issues in the particular network. To do so, the service repeatedly selects a candidate set of model settings based in part on the data lake of network telemetry data, trains a machine learning model using network telemetry data from the data lake that matches the candidate set of model settings, and tests performance of the trained model using an emulator that emulates network issues in the particular network. The service further deploys the generated machine learning model to the particular computer network for on-premise execution.

First claim

What is claimed is:

1. A method comprising:
maintaining, by a network assurance service, a data lake of network telemetry data obtained by the service from one or more computer networks;
generating, by the service, a machine learning model for on-premise execution in a particular computer network to detect network issues in the particular network by repeatedly:
selecting a candidate set of model settings based in part on the data lake of network telemetry data,
training a machine learning model using network telemetry data from the data lake that matches the candidate set of model settings, and
testing performance of the trained model using an emulator that emulates network issues in the particular network; and

deploying, by the service, the generated machine learning model to the particular computer network for on-premise execution.

2. The method as in claim 1 , wherein the candidate set of model settings is indicative of at least one of: a number of neural network layers for the machine learning model or a network telemetry parameter to be used as an input for the model.

3. The method as in claim 1 , wherein maintaining the data lake of network telemetry data obtained by the service from one or more computer networks comprises:
sending a compact representation of the data lake to a computer network that acts as a filter for network telemetry data exported by that network to the service.

4. The method as in claim 1 , wherein maintaining the data lake of network telemetry data obtained by the service from one or more computer networks comprises:
determining that a portion of the network telemetry data in the data lake is associated with a bug; and
filtering the portion of the telemetry data in the data lake associated with the bug from being used to train a machine learning model.

5. The method as in claim 4 , further comprising:
re-generating the machine learning model for on-premise execution in the particular computer network, after filtering the portion of the telemetry data in the data lake associated with the bug from being used to train a machine learning model; and
deploying the re-generated machine learning model to the particular computer network for on-premise execution.

6. The method as in claim 1 , wherein testing performance of the trained machine learning model using the emulator that emulates network issues in the particular network comprises:
assigning a score to the trained model based in part on a number of emulated network issues detected by the trained model.

7. The method as in claim 6 , wherein the assigned score is based further in part on feedback received from a user interface.

8. The method as in claim 6 , wherein generating the machine learning model for on-premise execution in the particular computer network comprises:
selecting the trained model for on-premise execution in the particular network based on its assigned score.

9. The method as in claim 1 , wherein the particular computer network is a wireless network and the network issues comprise at least one of: onboarding issues, throughput issues, authentication issues, or Dynamic Host Configuration Protocol (DHCP) issues.

10. An apparatus, comprising:
one or more network interfaces;
a processor coupled to the network interfaces and configured to execute one or more processes; and
a memory configured to store a process executable by the processor, the process when executed configured to:
maintain a data lake of network telemetry data obtained by the service from one or more computer networks;
generate a machine learning model for on-premise execution in a particular computer network to detect network issues in the particular network by repeatedly:
selecting a candidate set of model settings based in part on the data lake of network telemetry data,
training a machine learning model using network telemetry data from the data lake that matches the candidate set of model settings, and
testing performance of the trained model using an emulator that emulates network issues in the particular network; and

deploy the generated machine learning model to the particular computer network for on-premise execution.

11. The apparatus as in claim 10 , wherein the candidate set of model settings is indicative of at least one of: a number of neural network layers for the machine learning model or a network telemetry parameter to be used as an input for the model.

12. The apparatus as in claim 10 , wherein the apparatus maintains the data lake of network telemetry data obtained by the service from one or more computer networks by:
sending a compact representation of the data lake to a computer network that acts as a filter for network telemetry data exported by that network to the service.

13. The apparatus as in claim 10 , wherein the apparatus maintains the data lake of network telemetry data obtained by the service from one or more computer networks by:
determining that a portion of the network telemetry data in the data lake is associated with a bug; and
filtering the portion of the telemetry data in the data lake associated with the bug from being used to train a machine learning model.

14. The apparatus as in claim 13 , wherein the process when executed is further configured to:
re-generate the machine learning model for on-premise execution in the particular computer network, after filtering the portion of the telemetry data in the data lake associated with the bug from being used to train a machine learning model; and
deploy the re-generated machine learning model to the particular computer network for on-premise execution.

15. The apparatus as in claim 10 , wherein testing performance of the trained machine learning model using the emulator that emulates network issues in the particular network comprises:
assigning a score to the trained model based in part on a number of emulated network issues detected by the trained model.

16. The apparatus as in claim 10 , wherein the process when executed is further configured to:
filter telemetry data associated with a bug from being added to the data lake.

17. The apparatus as in claim 10 , wherein the apparatus generates the machine learning model for on-premise execution in the particular computer network by:
selecting the trained model for on-premise execution in the particular network based on its assigned score.

18. The apparatus as in claim 10 , wherein the particular computer network is a wireless network and the network issues comprise at least one of: onboarding issues, throughput issues, authentication issues, or Dynamic Host Configuration Protocol (DHCP) issues.

19. A tangible, non-transitory, computer-readable medium storing program instructions that cause a network assurance service to execute a process comprising:
maintaining, by the network assurance service, a data lake of network telemetry data obtained by the service from one or more computer networks;
generating, by the service, a machine learning model for on-premise execution in a particular computer network to detect network issues in the particular network by repeatedly:
selecting a candidate set of model settings based in part on the data lake of network telemetry data,
training a machine learning model using network telemetry data from the data lake that matches the candidate set of model settings, and
testing performance of the trained model using an emulator that emulates network issues in the particular network; and

deploying, by the service, the generated machine learning model to t

Assignees

• Cisco Tech Inc

Inventors

• Di Pietro Andrea

Classifications

• G06N3/09

  Supervised learning · CPC title

• G06N3/0985

  Hyperparameter optimisation; Meta-learning; Learning-to-learn · CPC title

• G06N3/0499

  Feedforward networks · CPC title

• G06N3/0895

  Weakly supervised learning, e.g. semi-supervised or self-supervised learning · CPC title

• H04L41/147

  for predicting network behaviour · CPC title