Method and apparatus for multi-frame prediction error-based video anomaly detection

Patent metadata

Field	Value
Publication number	US-12437381-B2
Application number	US-202218086795-A
Country	US
Kind code	B2
Filing date	Dec 22, 2022
Priority date	Feb 23, 2022
Publication date	Oct 7, 2025
Grant date	Oct 7, 2025

Abstract

Disclosed is a method and apparatus for multi-frame prediction error-based video anomaly detection. The disclosed apparatus for multi-frame prediction error-based video anomaly detection includes a multi-frame predictor configured to predict a current frame and adjacent frames using a prediction model; a multi-frame prediction error obtainer configured to obtain a multi-frame prediction error by obtaining a prediction error of the current frame and prediction errors of the adjacent frames that each prediction error is a difference between a predicted frame and a real frame and by adding the prediction error of the current frame and the prediction errors of the adjacent frames; and an anomaly score evaluator configured to evaluate an anomaly score based on the obtained multi-frame prediction error of the current frame.

First claim

What is claimed is:

1. A video anomaly detection apparatus comprising:
a multi-frame predictor configured to predict a current frame and adjacent frames using a prediction model;
a multi-frame prediction error obtainer configured to obtain a multi-frame prediction error by obtaining a prediction error of the current frame and prediction errors of the adjacent frames that each prediction error is a difference between a predicted frame and a real frame and by adding the prediction error of the current frame and the prediction errors of the adjacent frames; and
an anomaly score evaluator configured to evaluate an anomaly score based on the obtained multi-frame prediction error of the current frame,
wherein the anomaly score evaluator is configured to
obtain the anomaly score for the current frame by using a cascade sliding window method for the obtained multi-frame prediction error,
perform, when moving the window, a movement process for the window such that the window moves evenly in both directions, by moving the window from bottom center to bottom left of the multi-frame prediction error by the size of the window,
each time the window moves, obtain a mean value for a patch of the multi-frame prediction error corresponding to the window, when the window reaches the bottom left, reposition the window at the bottom center, then move the window to bottom right of the multi-frame prediction error by the size of the window, and when the window reaches the bottom right, reposition the window at the bottom center again and then move the window upward by the size of the window,
then decrease the size of the window by a predetermined amount, repeatedly perform the movement process for the window, and in response that the window reaches top right of the multi-frame prediction error, sort mean values for patches of the multi-frame prediction error corresponding to the window, obtained each time the window moves, in ascending order, select a predetermined number of patches from the sorted ascending mean values, and average the selected patches to use as the anomaly score, and
during the movement process for the window, move a x-coordinate of the window to x=the size of the window(s) in response to the window being unable to reach left side of the multi-frame prediction error due to insufficient remaining space of the multi-frame prediction error, move the x-coordinate of the window to x=a width of the frame (w)−the size of the window(s) in response to the window being unable to reach right side of the multi-frame prediction error due to insufficient remaining space of the multi-frame prediction error, and move a y-coordinate of the window to y=the size of the window(s) in response to the window being unable to reach top side of the multi-frame prediction error due to insufficient remaining space of the multi-frame prediction error.

2. The video anomaly detection apparatus of claim 1 , wherein, to predict the current frame and the adjacent frames, the multi-frame predictor is configured to predict the current frame and the adjacent frames by receiving and concatenating a previous frame and a next frame in color through two sub-networks for each of the current frame and the adjacent frames.

3. The video anomaly detection apparatus of claim 1 , wherein the multi-frame prediction error obtainer is configured to obtain prediction errors for the current frame and the adjacent frames by squaring a pixel difference between the predicted frame and the real frame for each frame and to obtain a multi-frame prediction error that is a sum of the prediction error of the current frame and the prediction errors of the adjacent frames to reduce an incorrect prediction error gap.

4. The video anomaly detection apparatus of claim 3 , wherein the multi-frame prediction error obtainer is configured to obtain a multi-frame prediction error for the current frame by multiplying the prediction error by a weight for reducing influence of a prediction error of a corresponding adjacent frame on anomaly detection of the current frame as a distance between the adjacent frame and the current frame increases and then summing each weight-multiplied prediction error.

5. The video anomaly detection apparatus of claim 4 , wherein an equation for calculating the multi-frame prediction error MPE i,j t is represented as follows:

MPE

i
,
j

t

=

∑

k
=

t
-
n

t
+
n

w

❘
"\[LeftBracketingBar]"

k
-
t

❘
"\[RightBracketingBar]"

·

PE

i
,
j

t

where t denotes a time, w denotes a weight, and PE i,j t denotes a prediction error.

6. A video anomaly detection method comprising:
predicting, by a multi-frame predictor, a current frame and adjacent frames using a prediction model;
obtaining, by a multi-frame prediction error obtainer, a multi-frame prediction error by obtaining a prediction error of the current frame and prediction errors of the adjacent frames that each prediction error is a difference between a predicted frame and a real frame and by adding the prediction error of the current frame and the prediction errors of the adjacent frames; and
evaluating, by an anomaly score evaluator, an anomaly score based on the obtained multi-frame prediction error of the current frame,
wherein the evaluating of the anomaly score comprises
obtaining the anomaly score for the current frame by using a cascade sliding window method for the obtained multi-frame prediction error,
performing, when moving the window, a movement process for the window such that the window moves evenly in both directions, by moving the window from bottom center to bottom left of the multi-frame prediction error by the size of the window,
each time the window moves, obtaining a mean value for a patch of the multi-frame prediction error corresponding to the window, when the window reaches the bottom left, repositioning the window at the bottom center, then moving the window to bottom right of the multi-frame prediction error by the size of the window, and when the window reaches the bottom right, repositioning the window at the bottom center again and then moving the window upward by the size of the window,
then decreasing the size of the window by a predetermined amount, repeatedly performing the movement process for the window, and in response that the window reaches top right of the multi-frame prediction error, sorting mean values for patches of the multi-frame prediction error corresponding to the window, obtained each time the window moves, in ascending o

Assignees

• Industry Academia Cooperation Group Of Sejong Univ

Inventors

• Kim Young-Gab
• Kim Yujun

Classifications

• G06T2207/10024

  Color image · CPC title

• G06T2207/30232

  Surveillance · CPC title

• G06T2207/30168

  Image quality inspection · CPC title

• G06T2207/10016

  Video; Image sequence · CPC title

• G06N3/04

  Architecture, e.g. interconnection topology · CPC title