Monitoring system, monitoring method, and computer program product

Patent metadata

Field	Value
Publication number	US-12032467-B2
Application number	US-201916542420-A
Country	US
Kind code	B2
Filing date	Aug 16, 2019
Priority date	Jan 15, 2019
Publication date	Jul 9, 2024
Grant date	Jul 9, 2024

Abstract

A monitoring system includes storage, and one or more processors. The storage stores at least one of first output data that is obtained from a learning model, or first statistical information that is obtained from the first output data. The processors calculate a degree of abnormality indicating a degree of change in statistical information of second output data with respect to the first statistical information, or a degree of change in the statistical information of the second output data with respect to second statistical information. The processors determine whether or not there is occurrence of an abnormality in the learning model, on the basis of the degree of abnormality. The processors output information indicating occurrence of the abnormality, in a case where occurrence of the abnormality is determined.

First claim

What is claimed is:

1. A monitoring system comprising:
storage configured to store at least a plurality of pieces of first output data that are obtained by inputting first input data in a learning model that is learned, or first statistical information that is obtained from the plurality of pieces of first output data;
one or more processors configured to:
train the learning model using the first input data using backpropagation;
acquire a plurality of pieces of second output data that are obtained by inputting second input data in the learning model;
calculate a degree of abnormality indicating at least one of a degree of change in second statistical information that is obtained from the plurality of pieces of second output data with respect to the first statistical information, or another degree of change in the second statistical information with respect to third statistical information that is calculated from a plurality of pieces of fourth statistical information including the first statistical information;
determine whether or not there is an occurrence of an abnormality in the learning model, based on the degree of abnormality;
output information indicating the occurrence of the abnormality, in a case where the occurrence of the abnormality is determined;
extract, in a case where the occurrence of the abnormality is determined, abnormal data that is at least one piece of the plurality of pieces of second output data that is a cause of the abnormality, among the plurality of pieces of second output data; and
output internal information indicating how the abnormal data is processed inside the learning model, the internal information including information visualizing a relationship between a boundary between classification classes and the abnormal data.

2. The monitoring system according to claim 1 , wherein
the learning model is a neural network.

3. The monitoring system according to claim 2 , wherein
the plurality of pieces of first output data and the plurality of pieces of second output data are log its or probability values that are output from the neural network.

4. The monitoring system according to claim 1 , wherein
the first statistical information is an arithmetic mean, a standard deviation, a median value, or a variance of the plurality of pieces of first output data.

5. The monitoring system according to claim 1 , wherein
the third statistical information is a mean value of the plurality of pieces of fourth statistical information.

6. The monitoring system according to claim 1 , wherein
the degree of change in the second statistical information with respect to the first statistical information is a Mahalanobis Distance or a Euclidean distance.

7. The monitoring system according to claim 1 , wherein
the processors determine there is the occurrence of the abnormality in the learning model when the degree of abnormality exceeds a threshold.

8. The monitoring system according to claim 1 , wherein
the plurality of pieces of first output data are output from the learning model to which the first input data is input, and
the plurality of pieces of second output data are output from the learning model to which the second input data is input.

9. A monitoring method comprising:
storing, in storage, at least a plurality of pieces of first output data that are obtained by inputting first input data in a learning model that is learned, or first statistical information that is obtained from the plurality of pieces of first output data;
training the learning model using the first input data using backpropagation;
acquiring a plurality of pieces of second output data that are obtained by inputting second input data in the learning model;
calculating a degree of abnormality indicating at least one of a degree of change in second statistical information that is obtained from the plurality of pieces of second output data with respect to the first statistical information, or another degree of change in the second statistical information with respect to third statistical information that is calculated from a plurality of pieces of fourth statistical information including the first statistical information;
determining whether or not there is an occurrence of an abnormality in the learning model, based on the degree of abnormality;
outputting information indicating the occurrence of the abnormality, in a case where the occurrence of the abnormality is determined;
extracting, in a case where the occurrence of the abnormality is determined, abnormal data that is at least one piece of the plurality of pieces of second output data that is a cause of the abnormality, among the plurality of pieces of second output data; and
outputting internal information indicating how the abnormal data is processed inside the learning model, the internal information including information visualizing a relationship between a boundary between classification classes and the abnormal data.

10. A computer program product having a non-transitory computer readable medium including programmed instructions, wherein the programmed instructions, when executed by a computer, cause the computer to perform:
storing, in storage, at least a plurality of pieces of first output data that are obtained by inputting first input data in a learning model that is learned, or first statistical information that is obtained from the plurality of pieces of first output data;
training the learning model using the first input data using backpropagation;
acquiring a plurality of pieces of second output data that are obtained by inputting second input data in the learning model;
calculating a degree of abnormality indicating at least one of a degree of change in second statistical information that is obtained from the plurality of pieces of second output data with respect to the first statistical information, or another degree of change in the second statistical information with respect to third statistical information that is calculated from a plurality of pieces of fourth statistical information including the first statistical information;
determining whether or not there is an occurrence of an abnormality in the learning model, based on the degree of abnormality;
outputting information indicating the occurrence of the abnormality, in a case where the occurrence of the abnormality is determined;
extracting, in a case where the occurrence of the abnormality is determined, abnormal data that is at least one piece of the plurality of pieces of second output data that is a cause of the abnormality, among the plurality of pieces of second output data; and
outputting internal information indicating how the abnormal data is processed inside the learning model, the internal information including information visualizing a relationship between a boundary between classification classes and the abnormal data.

Assignees

• Toshiba Kk
• Kioxia Corp

Inventors

• Kimura Mitsuhiro
• Takimoto Takahiro
• Sugimoto Akira
• Haruki Kosuke
• Ozawa Masahiro

Classifications

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06N3/043

  based on fuzzy logic, fuzzy membership or fuzzy inference, e.g. adaptive neuro-fuzzy inference systems [ANFIS] · CPC title

• G06F11/0766

  Error or fault reporting or storing · CPC title

• G06F11/302

  where the computing system component is a software system · CPC title