Image-based abnormal event detection

Patent metadata

Field	Value
Publication number	US-11900679-B2
Application number	US-202017104329-A
Country	US
Kind code	B2
Filing date	Nov 25, 2020
Priority date	Nov 26, 2019
Publication date	Feb 13, 2024
Grant date	Feb 13, 2024

Abstract

Methods and systems for image-based abnormal event detection are disclosed. An example method includes obtaining a sequential set of images captured by a camera; generating a set of observed features for each of the images; generating a set of predicted features based on a portion of the sets of observed features that excludes the set of observed features for a last image in the sequential set of images; determining that a difference between the set of predicted features and the set of observed features for the last image in the sequential set of images satisfies abnormal event criteria; and in response to determining that the difference between the set of predicted features and the set of observed features for the last image in the sequential set of images satisfies abnormal event criteria, classifying the set of sequential images as showing an abnormal event.

First claim

What is claimed is:

1. A method comprising:
obtaining a sequential set of images captured by a camera that includes one or more earlier images and a latter image captured after each of the one or more earlier images;
generating a first set of observed features for at least some of the one or more earlier images in the sequential set of images;
generating a second set of observed features for the latter image in the sequential set of images;
accessing stored data indicating features of objects classified as familiar objects;
identifying, from the first set of observed features, one or more observed features that satisfy similarity criteria for matching a feature of at least one of the familiar objects;
filtering the identified one or more observed features from the first set of observed features to generate an unfamiliar object subset of the first set of observed features that excludes the identified one or more observed features;
generating a set of predicted features for the latter image in the sequential set of images using a prediction model and the unfamiliar object subset of the first set of observed features for the one or more earlier images;
identifying, from the second set of observed features, one or more observed features that satisfy similarity criteria for matching a feature of at least one of the familiar objects;
filtering the identified one or more observed features from the second set of observed features to generate an unfamiliar object subset of the second set of the second set of observed features that excludes the identified one or more observed features;
determining that a difference between the set of predicted features for the latter image in the sequential set of images and the unfamiliar object subset of the second set of observed features for the latter image in the sequential set of images satisfies a threshold difference;
in response to determining that the difference between the set of predicted features for the latter image in the sequential set of images and the unfamiliar object subset of the second set of observed features for the latter image in the sequential set of images satisfies the threshold difference, determining that the difference between the set of predicted features for the latter image in the sequential set of images and the unfamiliar object subset of the second set of observed features for the latter image in the sequential set of images satisfies one or more abnormal event criteria; and
in response to determining that the difference between the set of predicted features for the latter image in the sequential set of images and the unfamiliar object subset of the second set of observed features for the latter image in the sequential set of images satisfies the one or more abnormal event criteria, classifying the sequential set of images as showing an abnormal event.

2. The method of claim 1 , comprising:
in response to classifying the set of sequential images as showing an abnormal event, performing one or more actions.

3. The method of claim 1 , comprising:
obtaining sensor data from one or more sensors; and
adjusting the threshold difference using the sensor data.

4. The method of claim 3 , wherein adjusting the threshold difference using the sensor data comprises:
assigning a weighting factor to the sensor data; and
adjusting the threshold difference based on the weighting factor.

5. The method of claim 1 , wherein generating the first set of observed features for at least some of the one or more earlier images comprises generating a vector of observed feature values.

6. The method of claim 5 , wherein generating the set of predicted features for the latter image in the sequential set of images comprises:
processing the vector of observed feature values for the one or more earlier images with the prediction model to obtain an output of the prediction model, wherein the output of the prediction model comprises a vector of predicted feature values for the latter image in the sequential set of images.

7. The method of claim 1 , comprising:
obtaining a second sequential set of images captured by the camera that is classified as showing a normal event, the second sequential set of images including one or more earlier images and a latter image captured after each of the one or more earlier images;
generating a third set of observed features for each at least some of the one or more earlier images in the second sequential set of images;
generating a fourth set of observed features for the latter image in the second sequential set of images;
providing, to the prediction model, the third set of observed features for the one or more earlier images in the second sequential set of images;
receiving, as output from the prediction model, a second set of predicted features for the latter image in the second sequential set of images;
comparing the output from the prediction model to the fourth set of observed features for the latter image in the second sequential set of images; and
based on comparing the output from the prediction model to the fourth set of observed features for the latter image in the second sequential set of images, adjusting parameters of the prediction model.

8. The method of claim 1 , comprising using sets of observed features for sequential sets of images that are each classified as showing a normal event to train the prediction model to predict features of latter images of additional sequential sets of images.

9. The method of claim 6 , wherein the difference between the set of predicted features for the latter image in the sequential set of images and the unfamiliar object subset of the second set of observed features for the latter image in the sequential set of images comprises a distance between the vector of observed feature values and the vector of predicted feature values.

10. The method of claim 1 , wherein the prediction model is a machine learning model comprising at least one of a neural network, a support vector machine, a classifier, a regression model, a reinforcement learning model, a clustering model, a decision tree, a random forest model, a genetic algorithm, a Bayesian model, or a Gaussian mixture model.

11. The method of claim 1 , wherein the sequential set of images comprises a plurality of consecutive image frames.

12. The method of claim 1 , wherein the sequential set of images comprises a plurality of image frames captured at periodic time intervals.

13. The method of claim 1 , comprising:
obtaining a different sequential set of images captured by the camera that includes one or more earlier images and a latter image captured after each of the one or more earlier images;
generating a third set of observed features for at least some of the one or more earlier images in the different sequential set of images;
generating a fourth set of observed features for the latter image in the different sequential set of images;
generating a set of predicted features for the latter image in the different sequential set of images using the prediction model and the third set of observed features;
determining that a difference between the set of predicted features for the latter image in the different sequential set of images and the fourth set of observed features for the latter image in the different sequential set of images does not satisfy abnormal event criteria; and
in response to determining that the difference between the set of predicted features for the latter image in the different sequential set of images and the fourth set of observed features for the latter image in the different sequential set of images does not satisfy abnormal event criteri

Assignees

• Objectvideo Labs Llc

Inventors

• Lee Jangwon
• Qian Gang
• Beach Allison
• Madden Donald Gerard

Classifications

• G06F21/554

  involving event detection and direct action · CPC title

• G06V20/46 Primary

  Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames · CPC title

• G06F18/22

  Matching criteria, e.g. proximity measures · CPC title

• G06V10/454

  Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN] · CPC title

• G06V10/761

  Proximity, similarity or dissimilarity measures · CPC title