System to mitigate against adversarial samples for ml and ai models

Patent metadata

Field	Value
Publication number	US-2020394466-A1
Application number	US-201916440962-A
Country	US
Kind code	A1
Filing date	Jun 13, 2019
Priority date	Jun 13, 2019
Publication date	Dec 17, 2020
Grant date	—

Abstract

Embodiments of the disclosure disclose a system to mitigate against adversarial input samples for machine learning (ML)/artificial intelligence (AI) models. According to one embodiment, a system receives a query from a client for a ML service. The system calculates a similarity score for the query based on a number of prior queries received from the client, the similarity score representing a similarity between the received query and the prior queries. The system determines that the query is an adversarial query in response to determining that the similarity score is above a predetermined threshold.

First claim

What is claimed is:

1 . A computer-implemented method for processing data in a trusted environment, the method comprising:
receiving a query from a client for a machine learning (ML) service to be served by a target ML model; calculating a similarity score for the query based on a plurality of queries previously received from the client, the similarity score representing a similarity between the received query and the prior queries; determining whether the similarity score is above a predetermine threshold; and determining that the query is an adversarial query in response to determining that the similarity score is above a predetermined threshold.

2 . The method of claim 1 , further comprising:
designating the client associated with the adversarial query as an adversarial client, in response to determining that the query is an adversarial query; and servicing subsequent queries of the adversarial client with an alternate ML model from a collection of ML models that have been trained together with the target ML model, instead of the target ML model, to prevent exploration of a blind spot of the target ML model.

3 . The method of claim 2 , further comprising blocking additional queries from the adversarial client.

4 . The method of claim 2 , wherein the alternate ML model is randomly chosen from the collection of ML models for each of the subsequent queries to obfuscate a confidence score returned to the client by the randomly chosen ML model.

5 . The method of claim 2 , wherein the collection of ML models were trained together with the target ML model but with a different parameter including a different epoch.

6 . The method of claim 1 , wherein the similarity score is calculated based on a distance between any two inputs for any two queries.

7 . The method of claim 6 , wherein, if the two inputs are images, the distance includes a count of different pixels between the two images.

8 . The method of claim 6 , wherein, if the two inputs are images, the distance includes a sum of differences in pixels between the two images.

9 . The method of claim 6 , wherein, if the two inputs comprise two images, the distance includes a root mean square of differences in pixels between the two images.

10 . A non-transitory machine-readable medium having instructions stored therein, which when executed by a processor, cause the processor to perform operations, the operations comprising:
receiving a query from a client for a machine learning (ML) service to be served by a target ML model; calculating a similarity score for the query based on a plurality of queries previously received from the client, the similarity score representing a similarity between the received query and the prior queries; determining whether the similarity score is above a predetermine threshold; and determining that the query is an adversarial query in response to determining that the similarity score is above a predetermined threshold.

11 . The non-transitory machine-readable medium of claim 10 , wherein the operation further comprising:
designating the client associated with the adversarial query as an adversarial client, in response to determining that the query is an adversarial query; and servicing subsequent queries of the adversarial client with an alternate ML model from a collection of ML models that are trained together with the target ML model, instead of the target ML model, to prevent exploration of a blind spot of the target ML model.

12 . The non-transitory machine-readable medium of claim 11 , wherein the operation further comprising blocking additional queries from the adversarial client.

13 . The non-transitory machine-readable medium of claim 11 , wherein the alternate ML model is randomly chosen from the collection of ML models for each of the subsequent queries to obfuscate a confidence score returned to the client by the randomly chosen ML model.

14 . The non-transitory machine-readable medium of claim 11 , wherein the collection of ML models is trained together with the target ML model but with a different parameter including a different epoch.

15 . The non-transitory machine-readable medium of claim 10 , wherein the similarity score is calculated based on a distance between any two inputs for any two queries.

16 . A data processing system, comprising:
a processor; and a memory coupled to the processor to store instructions, which when executed by the processor, cause the processor to perform operations, the operations including
receiving a query from a client for a machine learning (ML) service to be served by a target ML model,
calculating a similarity score for the query based on a plurality of queries previously received from the client, the similarity score representing a similarity between the received query and the prior queries,
determining whether the similarity score is above a predetermine threshold, and
determining that the query is an adversarial query in response to determining that the similarity score is above a predetermined threshold.

17 . The system of claim 16 , wherein the operation further comprising:
designating the client associated with the adversarial query as an adversarial client, in response to determining that the query is an adversarial query; and servicing subsequent queries of the adversarial client with an alternate ML model from a collection of ML models that are trained together with the target ML model, instead of the target ML model, to prevent exploration of a blind spot of the target ML model.

18 . The system of claim 17 , wherein the operation further comprising blocking additional queries from the adversarial client.

19 . The system of claim 17 , wherein the alternate ML model is randomly chosen from the collection of ML models for each of the subsequent queries to obfuscate a confidence score returned to the client by the randomly chosen ML model.

20 . The system of claim 17 , wherein the collection of ML models is trained together with the target ML model but with a different parameter including a different epoch.

Assignees

• Baidu Usa Llc

Inventors

• Jia Yunhan
• Zhong Zhenyu
• Zhang Yulong
• Wei Tao

Classifications

• G06N3/08 Primary

  Learning methods · CPC title

• G06F18/217

  Validation; Performance evaluation; Active pattern learning techniques · CPC title

• G06F18/22

  Matching criteria, e.g. proximity measures · CPC title

• G06N3/045 Primary

  Combinations of networks · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title