Method and system for training and validating machine learning in network environments

Patent metadata

Field	Value
Publication number	US-11301778-B2
Application number	US-201916359336-A
Country	US
Kind code	B2
Filing date	Mar 20, 2019
Priority date	Mar 21, 2018
Publication date	Apr 12, 2022
Grant date	Apr 12, 2022

Abstract

A system and method for training and validating ML algorithms in real networks, including: generating synthetic traffic and receiving it along with real traffic; aggregating the received traffic into network flows by using metadata and transforming them to generate a first dataset readable by the ML algorithm, comprising features defined by the metadata; labelling the traffic and selecting a subset of the features from the labelled dataset used in an iterative training to generate a trained model; filtering out a part of real traffic to obtain a second labelled dataset; and selecting a subset of features from the second labelled dataset used for validating the trained model by comparing predicted results for the trained model and the labels; repeating the steps with a different subset of features to generate another trained model until results are positive in terms of precision or accuracy.

First claim

The invention claimed is:

1. A method for training and validating machine learning algorithms in real network environments wherein real traffic is moving between an internal network and an external network, the method comprising:
(i) generating synthetic traffic;
(ii) receiving packets of traffic comprising the real traffic and the generated synthetic traffic;
(iii) aggregating the packets of the received traffic into network flows by using metadata;
(iv) transforming each network flow to generate a first dataset with a format readable by a machine learning algorithm, the first dataset comprising features defined by the metadata for each network flow;
(v) labelling the real traffic and the synthetic traffic to obtain a labelled dataset which comprises the first dataset and additional features defined as labels, wherein the labelling is done by matching features from the first dataset and features collected from the metadata, and assigning an unknown label if there is no match;
(vi) selecting a subset of the features from the labelled dataset to be used in an iterative training to generate a trained model;
(vii) filtering out a part of real traffic of the labelled dataset to obtain a second labelled dataset;
(viii) selecting a subset of features used in the trained model from the second labelled dataset to be used for validating the trained model;
(ix) comparing predicted results for the trained model and the labels obtained from labelling; and
(x) if the comparison is negative, repeating steps (v)-(ix) with a different subset of features to generate another trained model.

2. The method according to claim 1 , wherein the packets of the received traffic are aggregated by metadata extracted from the packets and selected from the group consisting of source IP address, destination IP address, source port, destination port, TCP sequence number, start date, end date, and any other value of the network packet.

3. The method according to claim 1 , wherein the packets of the received traffic are aggregated by metadata calculated and selected from the group consisting of counters of packets per flow, list of protocol flags, average size of bytes, and any other statistical traffic information derived from the aggregation of the packets belonging to the same network flow.

4. The method according to claim 1 , further comprising storing in a historic cache the aggregated network flow and including additional metadata from other previous related flows in a defined time window, the additional metadata being selected from the group consisting of number of identical flows, average number of packets per flow in the identical flows, time between identical flows, and any other features mathematically derived from the historic cache for a period of time in the past.

5. The method according to claim 1 , further comprising:
if the machine learning algorithm requires real traffic for training process, including a part of real traffic with the unknown label into the first dataset before selecting the features from the first dataset to be used in the iterative training; and
otherwise, removing the real traffic labelled with the unknown label from the first dataset.

6. The method according to claim 1 , further comprising removing the labels obtained from labelling which are not needed in training process if the machine learning algorithm is unsupervised, before selecting the features from the first dataset to be used in the iterative training.

7. The method according to claim 1 , wherein the step of comparing is performed in terms of precision, accuracy, or any other criteria defined by configurable parameters.

8. A system for training and validating machine learning algorithms in real network environments comprising an internal network and an external network providing real traffic through a network device, wherein the system comprises at least one processor and memory configured to perform functions via appropriately encoded instructions in interaction with the network device, by implementing functional components that include:
at least one generator of synthetic traffic;
a probe configured for:
receiving packets of traffic comprising the real traffic and the generated synthetic traffic through the network device;
aggregating the packets of the received traffic into network flows by using metadata; and
transforming each network flow to generate a first dataset with a format readable by a machine learning algorithm, the first dataset comprising features defined by the metadata for each network flow;

a train and validation module comprising:
a labeller configured for:
(i) labelling the real traffic and the synthetic traffic to obtain a labelled dataset which comprises the first dataset and additional features defined as labels, wherein the labelling is done by matching features from the first dataset and features collected from the metadata, and assigning an unknown label where there is no match; and

a feature extractor configured for:
(ii) selecting a subset of the features from the labelled dataset to be used in an iterative training to generate a trained model;
(iii) filtering out a part of real traffic of the labelled dataset to obtain a second labelled dataset;
(iv) selecting a subset of features used in the trained model from the second labelled dataset to be used for validating the trained model; and

the train and validation module being further configured for:
(v) comparing predicted results for the trained model and the labels obtained from labelling; and, if the comparison is negative, repeating steps (i)-(v) with a different subset of features to generate another trained model.

9. The system according to claim 8 , wherein the generator of synthetic traffic is selected from a synthetic user module and a synthetic server.

10. The system according to claim 8 , wherein the probe aggregates packets of the received traffic by metadata extracted from the packets and selected from source IP address, destination IP address, source port, destination port, TCP sequence number, start date and end date or any other value of the network packet.

11. The system according to claim 8 , wherein the probe aggregates packets of the received traffic by metadata calculated and selected from counters of packets per flow, list of protocol flags, average size of bytes and any other statistical traffic information derived from the aggregation of the packets belonging to the same network flow.

12. The system according to claim 8 , further comprising a historic cache for storing the aggregated network flow and including additional metadata from other previous related flows in a defined time window, the additional metadata being selected from number of identical flows, average number of packets per flow in the identical flows, time between identical flows, and any other features mathematically derived from the historic cache for a period of time in the past.

13. The system according to claim 8 , wherein the train and validation module is further configured for:
if the machine learning algorithm requires real traffic for training process, including a part of real traffic with the unknown label into the first dataset before selecting the features from the first dataset to be used in the iterative training;
otherwise, removing the real traffic labelled with the unknown label from the first dataset.

14. The system according to claim 8 , wherein the train and validation module is further configured for removing the labels obtained from labelling which are not needed in training process if the machine learning algorithm is unsupervised, before selecting the

Assignees

• Telefonica Sa

Inventors

• Pastor Perales Antonio
• Lopez Diego R
• Mozo Velasco Alberto
• Gomez Canaval Sandra

Classifications

• H04L41/14

  Network analysis or design · CPC title

• G06N20/00 Primary

  Machine learning · CPC title

• H04L41/16 Primary

  using machine learning or artificial intelligence · CPC title

• H04L43/026

  using flow identification · CPC title

• Y02D30/50

  in wire-line communication networks, e.g. low power modes or reduced link rate · CPC title