Safety monitor for image misclassification

Patent metadata

Field	Value
Publication number	US-2020380383-A1
Application number	US-201916424162-A
Country	US
Kind code	A1
Filing date	May 28, 2019
Priority date	May 28, 2019
Publication date	Dec 3, 2020
Grant date	—

Abstract

Systems, apparatuses, and methods for implementing a safety monitor framework for a safety-critical inference application are disclosed. A system includes a safety-critical inference application, a safety monitor, and an inference accelerator engine. The safety monitor receives an input image, test data, and a neural network specification from the safety-critical inference application. The safety monitor generates a modified image by adding additional objects outside of the input image. The safety monitor provides the modified image and neural network specification to the inference accelerator engine which processes the modified image and provides outputs to the safety monitor. The safety monitor determines the likelihood of erroneous processing of the original input image by comparing the outputs for the additional objects with a known good result. The safety monitor complements the overall fault coverage of the inference accelerator engine and covers faults only observable at the network level.

First claim

What is claimed is:

1 . A system comprising:
an inference accelerator engine; and a safety monitor configured to:
convey real image data from a safety-critical application to the inference accelerator engine;
determine which known good object image data to generate based at least in part on a probability of occurrence in real image data and/or a frequency of detection in previous images;
generate the known good object image data and convey the known good object image data to the inference accelerator engine; and
generate a confidence indicator based on an analysis of results produced by the inference acceleration engine classifying the known good object image data, wherein the confidence indicator represents a probability that the real image data was classified correctly by the inference acceleration engine;
wherein the system is configured to perform one or more corrective actions in response to the confidence indicator not meeting a threshold.

2 . The system as recited in claim 1 , wherein the safety monitor is further configured to:
convey only real image data from a safety-critical application to the inference accelerator engine while operating in a first mode; enter a second mode responsive to detecting a first condition; and responsive to entering the second mode, generate the known good object image data and convey the known good object image data to the inference accelerator engine.

3 . The system as recited in claim 2 , wherein the first condition comprises receiving a signal from the safety-critical application to enter the second mode, and wherein a first corrective action is terminating the safety-critical application.

4 . The system as recited in claim 1 , wherein the safety monitor is further configured to:
generate a modified image by combining the known good object image data with an input image, wherein the known good object image data comprises one or more given objects; and convey the modified image to the inference accelerator engine.

5 . The system as recited in claim 4 , wherein the safety monitor is further configured to:
analyze detected outputs of previous images to track a frequency of detection of various objects in the previous images; determine if any objects have both a probability of occurrence that is greater than a first threshold and a frequency of detection in previous images that is greater than a second threshold; and add one or more first objects to a next image responsive to determining that the one or more first objects have both a probability of occurrence that is greater than the first threshold and a frequency of detection in previous images that is greater than the second threshold.

6 . The system as recited in claim 4 , wherein the safety monitor is further configured to:
detect at least one known good object in test vector data; add the at least one known good object to extra space outside of original boundaries of the input image; and create the modified image from the input image and the extra space.

7 . The system as recited in claim 1 , wherein the safety monitor is configured to receive, from the safety-critical application, test data which indicates how the known good object image data should be classified by the inference accelerator engine.

8 . A method comprising:
conveying, by a safety monitor, real image data from a safety-critical application to an inference accelerator engine; determine, by the safety monitor, which known good object image data to generate based at least in part on a probability of occurrence in real image data and/or a frequency of detection in previous images; generating, by the safety monitor, the known good object image data and convey the known good object image data to the inference accelerator engine; generating, by the safety monitor, a confidence indicator based on an analysis of results produced by the inference acceleration engine classifying the known good object image data, wherein the confidence indicator represents a probability that the real image data was classified correctly by the inference acceleration engine; and performing, by the safety-critical application, one or more corrective actions in response to the confidence indicator not meeting a threshold.

9 . The method as recited in claim 8 , further comprising the safety monitor:
conveying only real image data from a safety-critical application to the inference accelerator engine while operating in a first mode; entering a second mode responsive to detecting a first condition; and responsive to entering the second mode, generating the known good object image data and conveying the known good object image data to the inference accelerator engine.

10 . The method as recited in claim 9 , wherein the first condition comprises receiving a signal from the safety-critical application to enter the second mode, and wherein a first corrective action is terminating the safety-critical application.

11 . The method as recited in claim 8 , further comprising:
generating, by the safety monitor, a modified image by combining the known good object image data with an input image, wherein the known good object image data comprises one or more given objects; and conveying the modified image to the inference accelerator engine.

12 . The method as recited in claim 11 , further comprising:
analyzing, by the safety monitor, detected outputs of previous images to track a frequency of detection of various objects in the previous images; determining, by the safety monitor, if any objects have both a probability of occurrence that is greater than a first threshold and a frequency of detection in previous images that is greater than a second threshold; and adding, by the safety monitor, one or more first objects to a next image responsive to determining that the one or more first objects have both a probability of occurrence that is greater than the first threshold and a frequency of detection in previous images that is greater than the second threshold.

13 . The method as recited in claim 11 , further comprising:
detecting, by the safety monitor, at least one known good object in test vector data; adding, by the safety monitor, the at least one known good object to extra space outside of original boundaries of the input image; and creating, by the safety monitor, the modified image from the input image and the extra space.

14 . The method as recited in claim 8 , further comprising receiving, by the safety monitor from the safety-critical application, test data which indicates how the known good object image data should be classified by the inference accelerator engine.

15 . An apparatus comprising:
a memory storing program instructions; and at least one processor coupled to the memory, wherein the program instructions are executable by the at least one processor to:
convey real image data from a safety-critical application to an inference accelerator engine;
determine which known good object image data to generate based at least in part on a probability of occurrence in real image data and/or a frequency of detection in previous images;
generate the known good object image data and convey the known good object image data to the inference accelerator engine;
generate a confidence indicator based on an analysis of results produced by the inference acceleration engine classifying the known good object image data, wherein the confidence indicator represents a probability that the real image data was classified correctly by the inference acceleration engine; and
perform one or more corrective actions in response to

Assignees

• Ati Technologies Ulc

Inventors

• Kwong Tung Chuen
• Chan Benjamin Koon Pan
• Porpino Sobreira Marques David
• Ip Clarence
• Yu Hung Wilson

Classifications

• G06V10/454 Primary

  Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN] · CPC title

• G06V10/82

  using neural networks · CPC title

• G06N5/04 Primary

  Inference or reasoning models · CPC title

• G06F18/24

  Classification techniques · CPC title

• G06N3/042

  Knowledge-based neural networks; Logical representations of neural networks · CPC title