Evaluation of anomaly detection algorithms using impersonation data derived from user data

Patent metadata

Field	Value
Publication number	US-10965696-B1
Application number	US-201715797065-A
Country	US
Kind code	B1
Filing date	Oct 30, 2017
Priority date	Oct 30, 2017
Publication date	Mar 30, 2021
Grant date	Mar 30, 2021

Abstract

Techniques are provided for evaluating anomaly detection algorithms using impersonation data derived from user transaction data. An exemplary method comprises obtaining transaction data of a given enterprise organization comprising transactions of a plurality of users; generating impersonation data by modifying one or more features of a subset of the transaction data of the given enterprise organization; classifying (i) at least a portion of the transaction data of the given enterprise organization, and (ii) at least a portion of the impersonation data using the anomaly detection algorithm of the given enterprise organization, wherein records of the impersonation data comprise a known classification; and evaluating a performance of the anomaly detection algorithm of the given enterprise organization by comparing the classification of records of the impersonation data by the anomaly detection algorithm with the known classification.

First claim

What is claimed is:

1. A method, comprising:
obtaining transaction data of a given enterprise organization comprising transactions of a plurality of users, wherein the transaction data comprises one or more of online transactions, login communications and attempts to access protected resources;
extracting one or more features from at least a subset of the transaction data of the given enterprise organization;
generating impersonation data by modifying one or more of the extracted features of the subset of the transaction data of the given enterprise organization from a first value for a given feature to a second value for the given feature to simulate a plurality of attack classes, wherein the extracted features of the transaction data that are modified are defined for each simulated attack class, wherein the generating the impersonation data comprises modifying a feature value of one or more of a user identifier feature and a location feature of a given transaction of a given user from the transaction data to a different feature value from the transaction data;
classifying, using at least one processing device, (i) at least a portion of the transaction data of the given enterprise organization, and (ii) at least a portion of the impersonation data, using an anomaly detection algorithm of the given enterprise organization, wherein records of the impersonation data comprise a known classification;
evaluating, using the at least one processing device, a performance of the anomaly detection algorithm of the given enterprise organization to identify at least one anomaly by comparing the classification of records of the impersonation data by the anomaly detection algorithm with the known classification for at least one of the plurality of simulated attack classes; and
initiating one or more remedial actions in response to the identified at least one anomaly.

2. The method of claim 1 , wherein the features of the transaction data that are modified are selected to simulate a given predefined attack scenario.

3. The method of claim 1 , wherein one or more modified features of a given impersonation record are based on the transaction data of one or more of the plurality of users of the given enterprise organization.

4. The method of claim 1 , wherein the evaluating comprises determining a risk score for one or more records in the impersonation data.

5. The method of claim 1 , further comprising the step of training the anomaly detection algorithm of the given enterprise organization using a different subset of the transaction data of the given enterprise organization.

6. The method of claim 1 , wherein the anomaly detection algorithm further comprises a supervised learning method that uses the known classification of the impersonation data to detect one or more of at least one feature and at least one pattern that corresponds to an impersonation attempt.

7. The method of claim 1 , wherein the known classification in the impersonation data provides an indication that the impersonation data corresponds to a false event.

8. A computer program product, comprising a non-transitory machine-readable storage medium having encoded therein executable code of one or more software programs, wherein the one or more software programs when executed by at least one processing device perform the following steps:
obtaining transaction data of a given enterprise organization comprising transactions of a plurality of users, wherein the transaction data comprises one or more of online transactions, login communications and attempts to access protected resources;
extracting one or more features from at least a subset of the transaction data of the given enterprise organization;
generating impersonation data by modifying one or more of the extracted features of the subset of the transaction data of the given enterprise organization from a first value for a given feature to a second value for the given feature to simulate a plurality of attack classes, wherein the extracted features of the transaction data that are modified are defined for each simulated attack class, wherein the generating the impersonation data comprises modifying a feature value of one or more of a user identifier feature and a location feature of a given transaction of a given user from the transaction data to a different feature value from the transaction data;
classifying, using the at least one processing device, (i) at least a portion of the transaction data of the given enterprise organization, and (ii) at least a portion of the impersonation data, using an anomaly detection algorithm of the given enterprise organization, wherein records of the impersonation data comprise a known classification;
evaluating, using the at least one processing device, a performance of the anomaly detection algorithm of the given enterprise organization to identify at least one anomaly by comparing the classification of records of the impersonation data by the anomaly detection algorithm with the known classification for at least one of the plurality of simulated attack classes; and
initiating one or more remedial actions in response to the identified at least one anomaly.

9. The computer program product of claim 8 , wherein the features of the transaction data that are modified are selected to simulate a given predefined attack scenario.

10. The computer program product of claim 8 , wherein one or more modified features of a given impersonation record are based on the transaction data of one or more of the plurality of users of the given enterprise organization.

11. The computer program product of claim 8 , wherein the anomaly detection algorithm further comprises a supervised learning method that uses the known classification of the impersonation data to detect one or more of at least one feature and at least one pattern that corresponds to an impersonation attempt.

12. The computer program product of claim 8 , wherein the known classification in the impersonation data provides an indication that the impersonation data corresponds to a false event.

13. An apparatus, comprising:
a memory; and
at least one processing device, coupled to the memory, operative to implement the following steps:
obtaining transaction data of a given enterprise organization comprising transactions of a plurality of users, wherein the transaction data comprises one or more of online transactions, login communications and attempts to access protected resources;
extracting one or more features from at least a subset of the transaction data of the given enterprise organization;
generating impersonation data by modifying one or more of the extracted features of the subset of the transaction data of the given enterprise organization from a first value for a given feature to a second value for the given feature to simulate a plurality of attack classes, wherein the extracted features of the transaction data that are modified are defined for each simulated attack class, wherein the generating the impersonation data comprises modifying a feature value of one or more of a user identifier feature and a location feature of a given transaction of a given user from the transaction data to a different feature value from the transaction data;
classifying, using the at least one processing device, (i) at least a portion of the transaction data of the given enterprise organization, and (ii) at least a portion of the impersonation data, using an anomaly detection algorithm of the given enterprise organization, wherein records of the impersonation data comprise a known classification;
evaluating, using the at least one processing device, a performance of the anomaly dete

Assignees

• Emc Ip Holding Co Llc

Inventors

• Amram Shay
• Zaslavsky Alex

Classifications

• G06N3/09

  Supervised learning · CPC title

• G06N3/0895

  Weakly supervised learning, e.g. semi-supervised or self-supervised learning · CPC title

• G06N3/08

  Learning methods · CPC title

• G06N20/00

  Machine learning · CPC title

• H04L63/1425 Primary

  Traffic logging, e.g. anomaly detection · CPC title