Detection of an adversarial backdoor attack on a trained model at inference time

Patent metadata

Field	Value
Publication number	US-11601468-B2
Application number	US-201916451110-A
Country	US
Kind code	B2
Filing date	Jun 25, 2019
Priority date	Jun 25, 2019
Publication date	Mar 7, 2023
Grant date	Mar 7, 2023

Abstract

Systems, computer-implemented methods, and computer program products that can facilitate detection of an adversarial backdoor attack on a trained model at inference time are provided. According to an embodiment, a system can comprise a memory that stores computer executable components and a processor that executes the computer executable components stored in the memory. The computer executable components can comprise a log component that records predictions and corresponding activation values generated by a trained model based on inference requests. The computer executable components can further comprise an analysis component that employs a model at an inference time to detect a backdoor trigger request based on the predictions and the corresponding activation values. In some embodiments, the log component records the predictions and the corresponding activation values from one or more layers of the trained model.

First claim

What is claimed is:

1. A system, comprising:
a memory that stores computer executable components; and
a processor that executes the computer executable components stored in the memory, wherein the computer executable components comprise:
a log component that records predictions and corresponding activation values generated by a trained model based on inference requests; and
an analysis component that employs a model at an inference time to detect a backdoor trigger request based on the predictions and the corresponding activation values, wherein the model is selected from a group consisting of a clustering model, an activation clustering model, a heuristic model, an outlier detector model, a trained outlier detector model, a local outlier factor model, a trained local outlier factor model, a novelty detector model, and a trained one class support vector machine model.

2. The system of claim 1 , wherein the log component records the predictions and the corresponding activation values from one or more layers of the trained model.

3. The system of claim 1 , wherein:
the trained model is selected from a second group consisting of a trained artificial intelligence model, a trained machine learning model, a trained deep learning model, and a trained neural network model.

4. The system of claim 1 , wherein the computer executable components further comprise:
a verification component that verifies authenticity of at least one of: one or more of the inference requests; one or more of the predictions; or one or more of the corresponding activation values.

5. The system of claim 1 , wherein the computer executable components further comprise:
a trainer component that trains the model based on at least one of: one or more of the inference requests; one or more of the predictions and one or more of the corresponding activation values; one or more verified inference requests; or one or more verified predictions and one or more verified corresponding activation values.

6. The system of claim 1 , wherein the computer executable components further comprise:
an intercept component that intercepts an inference request submitted to the trained model and extracts from the trained model at least one of a prediction or one or more corresponding activation values generated in at least one layer of the trained model based on the inference request.

7. The system of claim 1 , wherein the computer executable components further comprise:
an action component that deactivates the trained model based on a detected backdoor trigger request.

8. The system of claim 1 , wherein the analysis component employs the model at the inference time to detect the backdoor trigger request based on the predictions and the corresponding activation values to facilitate at least one of: improved backdoor trigger request detection accuracy of the model; or reduced computational cost of a processing unit associated with the model.

9. A computer-implemented method, comprising:
recording, by a system operatively coupled to a processor, predictions and corresponding activation values generated by a trained model based on inference requests; and
employing, by the system, a model at an inference time to detect a backdoor trigger request based on the predictions and the corresponding activation values, wherein the model is selected from a group consisting of a clustering model, an activation clustering model, a heuristic model, an outlier detector model, a trained outlier detector model, a local outlier factor model, a trained local outlier factor model, a novelty detector model, and a trained one class support vector machine model.

10. The computer-implemented method of claim 9 , wherein the recording comprises:
recording, by the system, the predictions and the corresponding activation values from one or more layers of the trained model.

11. The computer-implemented method of claim 9 , wherein:
the trained model is selected from a second group consisting of a trained artificial intelligence model, a trained deep learning model, and a trained neural network model.

12. The computer-implemented method of claim 9 , further comprising:
verifying, by the system, authenticity of at least one of: one or more of the inference requests; one or more of the predictions; or one or more of the corresponding activation values.

13. The computer-implemented method of claim 9 , further comprising:
training, by the system, the model based on at least one of: one or more of the inference requests; one or more of the predictions and one or more of the corresponding activation values;
one or more verified inference requests; or one or more verified predictions and one or more verified corresponding activation values.

14. The computer-implemented method of claim 9 , further comprising:
intercepting, by the system, an inference request submitted to the trained model; and
extracting, by the system, from the trained model at least one of a prediction or one or more corresponding activation values generated in at least one layer of the trained model based on the inference request.

15. The computer-implemented method of claim 9 , further comprising:
deactivating, by the system, the trained model based on a detected backdoor trigger request.

16. A computer program product facilitating detection of an adversarial backdoor attack on a trained model at inference time, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a processor to cause the processor to:
record, by the processor, predictions and corresponding activation values generated by a trained model based on inference requests; and
employ, by the processor, a model at an inference time to detect a backdoor trigger request based on the predictions and the corresponding activation values, wherein the model is selected from a group consisting of a clustering model, an activation clustering model, a heuristic model, an outlier detector model, a trained outlier detector model, a local outlier factor model, a trained local outlier factor model, a novelty detector model, and a trained one class support vector machine model.

17. The computer program product of claim 16 , wherein the program instructions are further executable by the processor to cause the processor to:
record, by the processor, the predictions and the corresponding activation values from one or more layers of the trained model.

18. The computer program product of claim 16 , wherein:
the trained model is selected from a second group consisting of a trained artificial intelligence model, a trained machine learning model, a trained deep learning model, and a trained neural network model.

19. The computer program product of claim 16 , wherein the program instructions are further executable by the processor to cause the processor to:
train, by the processor, the model based on at least one of: one or more of the inference requests; one or more of the predictions and one or more of the corresponding activation values; one or more verified inference requests; or one or more verified predictions and one or more verified corresponding activation values.

20. The computer program product of claim 16 , wherein the program instructions are further executable by the processor to cause the processor to:
intercept, by the processor, an inference request submitted to the trained model; and
extract, by the processor, from the trained model at least one of a

Assignees

• IBM

Inventors

• Baracaldo Angel Nathalie
• Zhou Yi
• Chen Bryant
• Anwar Ali
• Ludwig Heiko H

Classifications

• G06F21/55

  Detecting local intrusion or implementing counter-measures · CPC title

• H04L63/145 Primary

  the attack involving the propagation of malware through the network, e.g. viruses, trojans or worms · CPC title

• G06F18/23

  Clustering techniques · CPC title

• G06N5/04

  Inference or reasoning models · CPC title

• G06F18/214 Primary

  Generating training patterns; Bootstrap methods, e.g. bagging or boosting · CPC title