System to mitigate against adversarial samples for ML and AI models

Patent metadata

Field	Value
Publication number	US-11216699-B2
Application number	US-201916440962-A
Country	US
Kind code	B2
Filing date	Jun 13, 2019
Priority date	Jun 13, 2019
Publication date	Jan 4, 2022
Grant date	Jan 4, 2022

Abstract

Embodiments of the disclosure disclose a system to mitigate against adversarial input samples for machine learning (ML)/artificial intelligence (AI) models. According to one embodiment, a system receives a query from a client for a ML service. The system calculates a similarity score for the query based on a number of prior queries received from the client, the similarity score representing a similarity between the received query and the prior queries. The system determines that the query is an adversarial query in response to determining that the similarity score is above a predetermined threshold.

First claim

What is claimed is:

1. A computer-implemented method for processing data in a trusted environment, the method comprising:
receiving a query from a client for a machine learning (ML) service to be served by a target ML model;
calculating a similarity score for the query based on a plurality of queries previously received from the client, the similarity score representing a similarity between the received query and prior queries;
determining whether the similarity score is above a predetermine threshold;
determining that the query is an adversarial query in response to determining that the similarity score is above a predetermined threshold;
designating the client associated with the adversarial query as an adversarial client, in response to determining that the query is an adversarial query; and
servicing subsequent queries of the adversarial client with an alternate ML model from a collection of ML models that have been trained together with the target ML model, instead of the target ML model, to prevent exploration of a blind spot of the target ML model.

2. The method of claim 1 , further comprising blocking additional queries from the adversarial client.

3. The method of claim 1 , wherein the alternate ML model is randomly chosen from the collection of ML models for each of the subsequent queries to obfuscate a confidence score returned to the client by the randomly chosen ML model.

4. The method of claim 1 , wherein the collection of ML models were trained together with the target ML model but with a different parameter including a different epoch.

5. The method of claim 1 , wherein the similarity score is calculated based on a distance between any two inputs for any two queries.

6. The method of claim 5 , wherein, if the two inputs are images, the distance includes a count of different pixels between the two images.

7. The method of claim 5 , wherein, if the two inputs are images, the distance includes a sum of differences in pixels between the two images.

8. The method of claim 5 , wherein, if the two inputs comprise two images, the distance includes a root mean square of differences in pixels between the two images.

9. A non-transitory machine-readable medium having instructions stored therein, which when executed by a processor, cause the processor to perform operations, the operations comprising:
receiving a query from a client for a machine learning (ML) service to be served by a target ML model;
calculating a similarity score for the query based on a plurality of queries previously received from the client, the similarity score representing a similarity between the received query and prior queries;
determining whether the similarity score is above a predetermine threshold;
determining that the query is an adversarial query in response to determining that the similarity score is above a predetermined threshold;
designating the client associated with the adversarial query as an adversarial client, in response to determining that the query is an adversarial query; and
servicing subsequent queries of the adversarial client with an alternate ML model from a collection of ML models that are trained together with the target ML model, instead of the target ML model, to prevent exploration of a blind spot of the target ML model.

10. The non-transitory machine-readable medium of claim 9 , wherein the operations further comprising blocking additional queries from the adversarial client.

11. The non-transitory machine-readable medium of claim 9 , wherein the alternate ML model is randomly chosen from the collection of ML models for each of the subsequent queries to obfuscate a confidence score returned to the client by the randomly chosen ML model.

12. The non-transitory machine-readable medium of claim 9 , wherein the collection of ML models is trained together with the target ML model but with a different parameter including a different epoch.

13. The non-transitory machine-readable medium of claim 9 , wherein the similarity score is calculated based on a distance between any two inputs for any two queries.

14. A data processing system, comprising:
a processor; and
a memory coupled to the processor to store instructions, which when executed by the processor, cause the processor to perform operations, the operations including
receiving a query from a client for a machine learning (ML) service to be served by a target ML model,
calculating a similarity score for the query based on a plurality of queries previously received from the client, the similarity score representing a similarity between the received query and prior queries,
determining whether the similarity score is above a predetermine threshold,
determining that the query is an adversarial query in response to determining that the similarity score is above a predetermined threshold,
designating the client associated with the adversarial query as an adversarial client, in response to determining that the query is an adversarial query, and
servicing subsequent queries of the adversarial client with an alternate ML model from a collection of ML models that are trained together with the target ML model, instead of the target ML model, to prevent exploration of a blind spot of the target ML model.

15. The system of claim 14 , wherein the operations further comprising blocking additional queries from the adversarial client.

16. The system of claim 14 , wherein the alternate ML model is randomly chosen from the collection of ML models for each of the subsequent queries to obfuscate a confidence score returned to the client by the randomly chosen ML model.

17. The system of claim 14 , wherein the collection of ML models is trained together with the target ML model but with a different parameter including a different epoch.

Assignees

• Baidu Usa Llc

Inventors

• Jia Yunhan
• Zhong Zhenyu
• Zhang Yulong
• Wei Tao

Classifications

• G06N3/08 Primary

  Learning methods · CPC title

• G06F18/217

  Validation; Performance evaluation; Active pattern learning techniques · CPC title

• G06N3/045 Primary

  Combinations of networks · CPC title

• G06F18/22

  Matching criteria, e.g. proximity measures · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title