Pattern discovery from high dimensional telemetry data using machine learning in a network assurance service

Patent metadata

Field	Value
Publication number	US-10778566-B2
Application number	US-201815988084-A
Country	US
Kind code	B2
Filing date	May 24, 2018
Priority date	May 24, 2018
Publication date	Sep 15, 2020
Grant date	Sep 15, 2020

Abstract

In one embodiment, a network assurance service that monitors a plurality of networks subdivides telemetry data regarding devices located in the networks into subsets, wherein each subset is associated with a device type, time period, metric type, and network. The service summarizes each subset by computing distribution percentiles of metric values in the subset. The service identifies an outlier subset by comparing distribution percentiles that summarize the subsets. The service reports insight data regarding the outlier subset to a user interface. The service adjusts the subsets based in part on feedback regarding the insight data from the user interface.

First claim

What is claimed is:

1. A method comprising:
subdividing, by a network assurance service that monitors a plurality of networks operated by two or more organizations, telemetry data regarding devices of a plurality of different types located in the networks into subsets, wherein the telemetry data is subdivided having four dimensions including: a device type of the devices, a time period, a metric associated with the devices, and an organization of the devices;
summarizing, by the service, each subset by computing distribution percentiles of metric values in the subset;
identifying, by the service, an outlier subset by comparing the distribution percentiles that summarize the subsets;
reporting, by the service, insight data regarding the outlier subset to a user interface; and
adjusting, by the service, the subsets based in part on feedback regarding the insight data from the user interface.

2. The method as in claim 1 , wherein the type of the devices comprises at least one of: radio, network access point, or router.

3. The method as in claim 1 , wherein the insight data reports a physical location of the network associated with the outlier subset.

4. The method as in claim 1 , further comprising:
receiving, at the service and via the user interface, the feedback regarding the insight data, wherein the feedback is indicative of a perceived relevancy of the insight data to a user of the user interface.

5. The method as in claim 1 , wherein adjusting the subsets based in part on the feedback regarding the insight data comprises:
re-subdividing the telemetry data into new subsets, wherein the subsets and the new subsets differ by at least one of: time period or metric type.

6. The method as in claim 1 , wherein adjusting the subsets based in part on the feedback regarding the insight data comprises:
re-subdividing the telemetry data into new subsets, to find a dimensional split for the telemetry data that results in insight data that receives positive feedback from the user interface.

7. The method as in claim 1 , wherein summarizing each subset by computing distribution percentiles of metric values in the subset comprises:
applying, by the service, differential privacy to the subsets based on their associated organizations so as to conceal information relating to a first organization of the two or more organizations from an end user of a second organization of the two or more organizations.

8. The method as in claim 1 , wherein identifying the outlier subset by comparing distribution percentiles that summarize the subsets comprises:
computing, by the service, Kolmogorov-Smirnov statistics between the subsets.

9. An apparatus, comprising:
one or more network interfaces to communicate with a network;
a processor coupled to the network interfaces and configured to execute one or more processes; and
a memory configured to store a process executable by the processor, the process when executed configured to:
subdivide telemetry data regarding devices of a plurality of different types located in a plurality of networks into subsets, wherein the plurality of networks are operated by two or more organizations, and wherein the telemetry data is subdivided having four dimensions including: a type of the devices, a time period, a metric associated with the devices, and an organization of the devices;
summarize each subset by computing distribution percentiles of metric values in the subset;
identify an outlier subset by comparing the distribution percentiles that summarize the subsets;
report insight data regarding the outlier subset to a user interface; and
adjust the subsets based in part on feedback regarding the insight data from the user interface.

10. The apparatus as in claim 9 , wherein the type of the devices comprises at least one of: radio, network access point, or router.

11. The apparatus as in claim 9 , wherein the insight data reports a physical location of the network associated with the outlier subset.

12. The apparatus as in claim 9 , wherein the process when executed is further configured to:
receive, via the user interface, the feedback regarding the insight data, wherein the feedback is indicative of a perceived relevancy of the insight data to a user of the user interface.

13. The apparatus as in claim 9 , wherein the apparatus adjusts the subsets based in part on the feedback regarding the insight data by:
re-subdividing the telemetry data into new subsets, wherein the subsets and the new subsets differ by at least one of: time period or metric type.

14. The apparatus as in claim 9 , wherein the apparatus adjusts the subsets based in part on the feedback regarding the insight data by:
re-subdividing the telemetry data into new subsets, to find a dimensional split for the telemetry data that results in insight data that receives positive feedback from the user interface.

15. The apparatus as in claim 9 , wherein the apparatus summarizes each subset by computing distribution percentiles of metric values in the subset by:
applying differential privacy to the subsets based on their associated organizations so as to conceal information relating to a first organization of the two or more organizations from an end user of a second organization of the two or more organizations.

16. The apparatus as in claim 9 , wherein the apparatus identifies the outlier subset by comparing distribution percentiles that summarize the subsets by:
computing Kolmogorov-Smirnov statistics between the subsets.

17. A tangible, non-transitory, computer-readable medium storing program instructions that cause a network assurance service that monitors a plurality of networks to execute a process comprising:
subdividing, by the network assurance service, telemetry data regarding devices of a plurality of different types located in the networks into subsets, wherein the plurality of networks are operated by two or more organizations, and wherein the telemetry data is subdivided having four dimensions including: a type of the devices, a time period, a metric associated with the devices, and an organization of the devices;
summarizing, by the service, each subset by computing distribution percentiles of metric values in the subset;
identifying, by the service, an outlier subset by comparing the distribution percentiles that summarize the subsets;
reporting, by the service, insight data regarding the outlier subset to a user interface; and
adjusting, by the service, the subsets based in part on feedback regarding the insight data from the user interface.

18. The computer-readable medium as in claim 17 , wherein adjusting the subsets based in part on the feedback regarding the insight data comprises:
re-subdividing the telemetry data into new subsets, wherein the subsets and the new subsets differ by at least one of: time period or metric type.

Assignees

• Cisco Tech Inc

Inventors

• Mermoud Grégory
• Vasseur Jean-Philippe
• Di Pietro Andrea

Classifications

• H04L45/02

  Topology update or discovery · CPC title

• H04L45/08 Primary

  Learning-based routing, e.g. using neural networks or artificial intelligence · CPC title

• H04L41/0803 Primary

  Configuration setting · CPC title

• H04L61/5014

  using dynamic host configuration protocol [DHCP] or bootstrap protocol [BOOTP] · CPC title

• H04L12/4641

  Virtual LANs, VLANs, e.g. virtual private networks [VPN] (LAN interconnection over a bridge based backbone H04L12/462; encapsulation techniques H04L12/4633; routing of packets H04L45/00; packet switches H04L49/00; virtual private networks for security H04L63/0272) · CPC title