Managing artificial intelligence models to identify goals of malicious attackers

Patent metadata

Field	Value
Publication number	US-12489798-B2
Application number	US-202318459126-A
Country	US
Kind code	B2
Filing date	Aug 31, 2023
Priority date	Aug 31, 2023
Publication date	Dec 2, 2025
Grant date	Dec 2, 2025

Abstract

Methods and systems for managing an artificial intelligence (AI) model are disclosed. An AI model may be part of an evolving AI model pipeline, the processes of which may include obtaining training data from data sources used to update the AI model. An attacker may introduce poisoned training data via one or more of the data sources as a form of attack on the AI model. When the poisoned training data is identified, the poisoned training data may be compared to existing training data to determine the attacker's goal. Based on the attacker's goal, remedial actions may be performed that may update operation of pipeline. The updated operation of the pipeline may reduce the computational expense for remediating impact of the poisoned training data, and may reduce the likelihood of obtaining poisoned training data in the future.

First claim

What is claimed is:

1 . A method for managing an artificial intelligence (AI) model, comprising:
determining that a portion of new training data for the AI model contains poisoned training data, the AI model being part of an evolving AI model pipeline; and after determining that the portion of new training data contains the poisoned training data:
identifying a portion of existing training data on which the portion of the new training data that has been determined as containing the poisoned training data is based;
for a sample of the portion of the new training data that has been determined as containing the poisoned training data, identifying at least one sample of the existing training data upon which the sample of the portion of the new training data is based;
identifying at least one label for the at least one sample;
identifying a goal of a malicious entity using the at least one label for the at least one sample; and
performing a remedial action set to update operation of the evolving AI model pipeline based on the goal.

2 . The method of claim 1 , wherein identifying the at least one sample of the existing training data upon which the sample of the portion of the new training data that has been determined as containing the poisoned training data is based comprises:
for the sample of the portion of the new training data that has been determined as containing the poisoned training data:
identifying a first sample of the existing training data that comprises a portion that is substantially similar to a first portion of the sample of the portion of the new training data that has been determined as containing the poisoned training data.

3 . The method of claim 2 , wherein identifying the at least one sample of the existing training data upon which the sample of the portion of the new training data is based further comprises:
further for the sample of the portion of the new training data:
identifying a second sample of the existing training data that comprises a portion that is substantially similar to a second portion of the sample of the portion of the new training data.

4 . The method of claim 3 , wherein the first portion of the sample of the portion of the new training data that has been determined as containing the poisoned training data is a first portion of a first image, and the second portion of the sample of the portion of the new training data that has been determined as containing the poisoned training data is a second portion of the first image.

5 . The method of claim 3 , wherein the first sample is a second image, the second sample is a third image, and a first label of the at least one label indicates a first depiction in the second image and a second label of the at least one label indicates a second depiction in the third image.

6 . The method of claim 5 , wherein identifying the goal of the malicious entity comprises:
classifying the goal based on a level of matching of the first label and the second label.

7 . The method of claim 6 , wherein classifying the goal comprises:
comparing the level of matching of the first label and the second label to a threshold; in a first instance of the comparing where the level of matching exceeds the threshold:
concluding that the goal is to reinforce an existing trend in the existing training data; and
in a second instance of the comparing where the level of matching does not exceed the threshold:
concluding that the goal is to dilute an existing trend in the existing training data.

8 . The method of claim 1 , wherein the evolving AI model pipeline provides inferences to inference consumers using AI models, and the new training data was scheduled for use in updating the AI models prior to the identification being made.

9 . A non-transitory machine-readable medium having instructions stored therein, which when executed by a processor, cause the processor to perform operations for managing an artificial intelligence (AI) model, the operations comprising:
determining that a portion of new training data for the AI model is poisoned training data, the AI model being part of an evolving AI model pipeline; and after determining that the portion of new training data contains the poisoned training data:
identifying a portion of existing training data on which the portion of the new training data that has been determined as containing the poisoned training data is based;
for a sample of the portion of the new training data that has been determined as containing the poisoned training data, identifying at least one sample of the existing training data upon which the sample of the portion of the new training data is based;
identifying at least one label for the at least one sample;
identifying a goal of a malicious entity using the at least one label for the at least one sample; and
performing a remedial action set to update operation of the evolving AI model pipeline based on the goal.

10 . The non-transitory machine-readable medium of claim 9 , wherein identifying the at least one sample of the existing training data upon which the sample of the portion of the new training data that has been determined as containing the poisoned training data is based comprises:
for the sample of the portion of the new training data that has been determined as containing the poisoned training data:
identifying a first sample of the existing training data that comprises a portion that is substantially similar to a first portion of the sample of the portion of the new training data that has been determined as containing the poisoned training data; and
identifying a second sample of the existing training data that comprises a portion that is substantially similar to a second portion of the sample of the portion of the new training data that has been determined as containing the poisoned training data.

11 . The non-transitory machine-readable medium of claim 10 , wherein the first portion of the sample of the portion of the new training data that has been determined as containing the poisoned training data is a first portion of a first image, and the second portion of the sample of the portion of the new training data that has been determined as containing the poisoned training data is a second portion of the first image.

12 . The non-transitory machine-readable medium of claim 11 , wherein the first sample is a second image, the second sample is a third image, and a first label of the at least one label indicates a first depiction in the second image and a second label of the at least one label indicates a second depiction in the third image.

13 . The non-transitory machine-readable medium of claim 12 , wherein identifying the goal of the malicious entity comprises:
classifying the goal based on a level of matching of the first label and the second label.

14 . The non-transitory machine-readable medium of claim 13 , wherein classifying the goal comprises:
comparing the level of matching of the first label and the second label to a threshold; in a first instance of the comparing where the level of matching exceeds the threshold:
concluding that the goal is to reinforce an existing trend in the existing training data; and
in a second instance of the comparing where the level of matching does not exceed the threshold:
concluding that the goal is to dilute an existing trend in the existing training data.

15 . The non-transitory machine-readable medium of claim 9 , wherein the evolving AI model pipeline provides inferences to inference consumers using AI models, and the new training data was scheduled for use in u

Assignees

• Dell Products Lp

Inventors

• Ezrielev Ofir
• Kushnir Tomer
• Savir Amihai

Classifications

• H04L63/1491 Primary

  using deception as countermeasure, e.g. honeypots, honeynets, decoys or entrapment · CPC title

• H04L2463/145

  Detection or countermeasures against cache poisoning · CPC title

• H04L63/205 Primary

  involving negotiation or determination of the one or more network security mechanisms to be used, e.g. by negotiation between the client and the server or between peers or by selection according to the capabilities of the entities involved (negotiation of communication capabilities H04L69/24) · CPC title