System and method for managing AI models based on downstream use of inferences

Patent metadata

Field	Value
Publication number	US-12536338-B2
Application number	US-202218147748-A
Country	US
Kind code	B2
Filing date	Dec 29, 2022
Priority date	Dec 29, 2022
Publication date	Jan 27, 2026
Grant date	Jan 27, 2026

Abstract

Methods and systems for identifying a goal of a malicious party that poisoned an artificial intelligence (AI) model are disclosed. As AI models are updated over time using new training data, snapshots of the AI models may be obtained. The snapshots may include information regarding the structure of the AI model, the inferences obtained from the AI model (e.g., outcomes of the inferences on inference consumers), and/or the training data used to train the AI model instance (e.g., data sources that supply the training data). A malicious party may introduce poisoned training data to an AI model to control the outcomes of consumed inferences on inference consumers. To identify the goal of the malicious party and/or to identify the malicious party itself, the relationships of information from the snapshots and the outcomes for inference consumer groups may be analyzed.

First claim

What is claimed is:

1 . A method for managing an artificial intelligence (AI) model, comprising:
making an identification that a training dataset is poisoned; identifying a tainted AI model instance that was obtained using the poisoned training dataset and the AI model; identifying a poisoned inference generated by the tainted AI model instance that has already been provided to an inference consumer; identifying an effect of the poisoned inference on the inference consumer that received the poisoned inference; and inferring, based on the effect, a goal of a malicious party by at least:
identifying membership of the inference consumer in a first group of inference consumers disproportionately impacted by the effect when compared to an impact of the effect on a second group of inference consumers;
identifying an inference consumer of the second group of inference consumers that modified an activity prior to use of the poisoned inference by the first group of inference consumers, the modification of the activity moving the inference consumer from the first group of inference consumers to the second group of inference consumers; and
identifying the goal based on the inference consumer of the second group of inference consumers that modified the activity.

2 . The method of claim 1 , wherein inferring the goal of the malicious party further comprises:
identifying a commonality among the first group of inference consumers; and identifying the goal further based on the commonality.

3 . The method of claim 2 , wherein the first group of inference consumers are participants in a market for a product, and the effect of the poisoned inference was a change in price of the product used by the participants.

4 . The method of claim 1 , wherein the activity is sale of a product at a first price, and the modified activity is sale of the product at a second price.

5 . The method of claim 1 , further comprising identifying a data source, the data source being used by the malicious party to provide the poisoned training dataset.

6 . The method of claim 5 , further comprising establishing a level of scrutiny for new training data being collected from the data source, the level of scrutiny being related to a level of impact of the poisoned inference.

7 . The method of claim 5 , further comprising eliminating the data source as a source for new training data collection.

8 . The method of claim 1 , wherein the effect of the poisoned inference on the inference consumer is identified based on a report of the effect, the report being received from the inference consumer to which the poisoned inference was provided.

9 . The method of claim 1 , wherein the tainted AI model instance is already deployed and providing inferences to inference consumers when the tainted AI model instance is identified using the poisoned training dataset and the AI model, the poisoned inference being one of the inferences and the inference consumer being one of the inference consumers.

10 . A non-transitory machine-readable medium having instructions stored therein, which when executed by a processor, cause the processor to perform operations for managing an artificial intelligence (AI) model, the operations comprising:
making an identification that a training dataset is poisoned; identifying a tainted AI model instance that was obtained using the poisoned training dataset and the AI model; identifying a poisoned inference generated by the tainted AI model instance that has already been provided to an inference consumer; identifying an effect of the poisoned inference on the inference consumer that received the poisoned inference; and inferring, based on the effect, a goal of a malicious party by at least:
identifying membership of the inference consumer in a first group of inference consumers disproportionately impacted by the effect when compared to an impact of the effect on a second group of inference consumers;
identifying an inference consumer of the second group of inference consumers that modified an activity prior to use of the poisoned inference by the first group of inference consumers, the modification of the activity moving the inference consumer from the first group of inference consumers to the second group of inference consumers; and
identifying the goal based on the inference consumer of the second group of inference consumers that modified the activity.

11 . The non-transitory machine-readable medium of claim 10 , wherein inferring the goal of the malicious party further comprises:
identifying a commonality among the first group of inference consumers; and identifying the goal further based on the commonality.

12 . The non-transitory machine-readable medium of claim 11 , wherein the first group of inference consumers are participants in a market for a product, and the effect of the poisoned inference was a change in price of the product used by the participants.

13 . The non-transitory machine-readable medium of claim 10 , wherein the activity is sale of a product at a first price, and the modified activity is sale of the product at a second price.

14 . The non-transitory machine-readable medium of claim 10 , wherein the operations further comprise identifying a data source, the data source being used by the malicious party to provide the poisoned training dataset.

15 . A data processing system, comprising:
a processor; and a memory coupled to the processor to store instructions, which when executed by the processor, cause the processor to perform operations for managing an artificial intelligence (AI) model, the operations comprising:
making an identification that a training dataset is poisoned,
identifying a tainted AI model instance that was obtained using the poisoned training dataset and the AI model,
identifying a poisoned inference generated by the tainted AI model instance that has already been provided to an inference consumer;
identifying an effect of the poisoned inference on the inference consumer that received the poisoned inference, and
inferring, based on the effect, a goal of a malicious party by at least:
identifying membership of the inference consumer in a first group of inference consumers disproportionately impacted by the effect when compared to an impact of the effect on a second group of inference consumers;
identifying an inference consumer of the second group of inference consumers that modified an activity prior to use of the poisoned inference by the first group of inference consumers, the modification of the activity moving the inference consumer from the first group of inference consumers to the second group of inference consumers; and
identifying the goal based on the inference consumer of the second group of inference consumers that modified the activity.

16 . The data processing system of claim 15 , wherein inferring the goal of the malicious party further comprises:
identifying a commonality among the first group of inference consumers; and identifying the goal further based on the commonality.

17 . The data processing system of claim 16 , wherein the first group of inference consumers are participants in a market for a product, and the effect of the poisoned inference was a change in price of the product used by the participants.

18 . The data processing system of claim 15 , wherein the activity is sale of a product at a first price, and the modified activity is sale of the product at a second price.

19 . The data processing system of claim 15 , wherein th

Assignees

• Dell Products Lp

Inventors

• Ezrielev Ofir
• Savir Amihai
• Kushnir Tomer

Classifications

• G06Q30/0201

  Market modelling; Market analysis; Collecting market data · CPC title

• G06F21/64 Primary

  Protecting data integrity, e.g. using checksums, certificates or signatures · CPC title