Detecting adversarial samples by a vision based perception system

Patent metadata

Field	Value
Publication number	US-2020394472-A1
Application number	US-201916440973-A
Country	US
Kind code	A1
Filing date	Jun 13, 2019
Priority date	Jun 13, 2019
Publication date	Dec 17, 2020
Grant date	—

Abstract

In one embodiment, a system receives a first image captured by a capturing device of an ADV. The system applies an image transformation to the first image to generate a second image. The system applies an object detection model to the first and the second images to generate a first and a second output, respectively. The system calculates a similarity metric between the first and the second output. The system detects the first image as an adversarial sample if a temporal variation of the similarity metric between the first image and a prior image is above a threshold.

First claim

What is claimed is:

1 . A computer-implemented method, comprising:
receiving a first image captured by a capturing device of an autonomous driving vehicle (ADV); performing an image transformation to transform the first image to a second image; applying an object detection model to the first image and the second image to generate a first output and a second output, respectively; calculating a similarity metric between the first output and the second output; and detecting that the first image as an adversarial sample if a temporal variation of the similarity metric between the first image and a prior image is above a predetermined threshold.

2 . The method of claim 1 , wherein each of the first output and the second output includes a list of bounding boxes, the locations of the bounding boxes, and annotation of each class objects for the bounding boxes for the input image.

3 . The method of claim 1 , wherein the image transformation includes a color depth reduction, an image compression, or a blurring transformation.

4 . The method of claim 1 , wherein the similarity metric is calculated based on a distance between a plurality of inputs.

5 . The method of claim 4 , wherein the distance includes differences in class prediction, number of bounding boxes, and overlapping regions of the bounding boxes.

6 . The method of claim 1 , further comprising activating a failsafe mechanism for the ADV if an adversarial sample is detected.

7 . The method of claim 6 , wherein the failsafe mechanism includes ignoring the adversarial sample or transferring control to a user of the ADV if the ADV is in self-driving mode.

8 . A non-transitory machine-readable medium having instructions stored therein, which when executed by a processor, cause the processor to perform operations, the operations comprising:
receiving a first image captured by a capturing device of an autonomous driving vehicle (ADV); performing an image transformation to transform the first image to a second image; applying an object detection model to the first image and the second image to generate a first output and a second output, respectively; calculating a similarity metric between the first output and the second output; and detecting that the first image as an adversarial sample if a temporal variation of the similarity metric between the first image and a prior image is above a predetermined threshold.

9 . The non-transitory machine-readable medium of claim 8 , wherein each of the first output and the second output includes a list of bounding boxes, the locations of the bounding boxes, and annotation of each class objects for the bounding boxes for the input image.

10 . The non-transitory machine-readable medium of claim 8 , wherein the image transformation includes a color depth reduction, an image compression, or a blurring transformation.

11 . The non-transitory machine-readable medium of claim 8 , wherein the similarity metric is calculated based on a distance between a plurality of inputs.

12 . The non-transitory machine-readable medium of claim 11 , wherein the distance includes differences in class prediction, number of bounding boxes, and overlapping regions of the bounding boxes.

13 . The non-transitory machine-readable medium of claim 8 , wherein the operations further comprise activating a failsafe mechanism for the ADV if an adversarial sample is identified.

14 . The non-transitory machine-readable medium of claim 13 , wherein the failsafe mechanism includes ignoring the adversarial sample or transferring control to a user of the ADV if the ADV is in self-driving mode.

15 . A data processing system, comprising:
a processor; and a memory coupled to the processor to store instructions, which when executed by the processor, cause the processor to perform operations, the operations including receiving a first image captured by a capturing device of an autonomous driving vehicle (ADV),
performing an image transformation to transform the first image to a second image,
applying an object detection model to the first image and the second image to generate a first output and a second output, respectively,
calculating a similarity metric between the first output and the second output, and
detecting that the first image as an adversarial sample if a temporal variation of the similarity metric between the first image and a prior image is above a predetermined threshold.

16 . The system of claim 15 , wherein each of the first output and the second output includes a list of bounding boxes, the locations of the bounding boxes, and annotation of each class objects for the bounding boxes for the input image.

17 . The system of claim 15 , wherein the image transformation includes a color depth reduction, an image compression, or a blurring transformation.

18 . The system of claim 15 , wherein the similarity metric is calculated based on a distance between a plurality of inputs.

19 . The system of claim 18 , wherein the distance includes differences in class prediction, number of bounding boxes, and overlapping regions of the bounding boxes.

20 . The system of claim 15 , wherein the operations further comprise activating a failsafe mechanism for the ADV if an adversarial sample is identified.

21 . The system of claim 20 , wherein the failsafe mechanism includes ignoring the adversarial sample or transferring control to a user of the ADV if the ADV is in self-driving mode.

Assignees

• Baidu Usa Llc

Inventors

• Jia Yunhan
• Zhong Zhenyu
• Zhang Yulong
• Wei Tao

Classifications

• G06V30/248

  involving plural approaches, e.g. verification by template match; Resolving confusion among similar patterns, e.g. "O" versus "Q" (G06V30/242 takes precedence) · CPC title

• G06F18/2193

  based on specific statistical tests · CPC title

• G06V20/56 Primary

  exterior to a vehicle by using sensors mounted on the vehicle · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G06F18/241

  relating to the classification model, e.g. parametric or non-parametric approaches · CPC title