Processing of surveillance video streams using image classification and object detection

Patent metadata

Field	Value
Publication number	US-2022374635-A1
Application number	US-202117326628-A
Country	US
Kind code	A1
Filing date	May 21, 2021
Priority date	May 21, 2021
Publication date	Nov 24, 2022
Grant date	—

Abstract

Systems and methods for processing surveillance video streams using image classification and object detection are described. Video data from a video image sensor may be processed using an image classifier to determine whether an object type is present in a video frame. If the object type is present, the video frame and/or subsequent video frames may be processed using an object detector to provide additional object data, such as position information, for use in other video surveillance processes. In some examples, an event message may be generated and sent to a video surveillance application in response to selective object detection.

First claim

1 . A system, comprising:
a video image sensor; and a controller configured to:
receive video data from the video image sensor, wherein the video data includes a time-dependent video stream of video frames captured by the video image sensor;
determine, using an image classifier, whether a first object type is present in a first frame of the video data;
determine, using an object detector, position information for a detected object in the video data having the first object type; and
send, over a network, an event notification to a video surveillance application.

2 . The system of claim 1 , wherein:
the controller comprises:
a processor;
a memory;
the image classifier, stored in the memory for execution by the processor, and configured to use:
a first set of processor resources; and
a first set of memory resources; and

the object detector, stored in the memory for execution by the processor, and configured to use:
a second set of processor resources; and
a second set of memory resources;

the first set of processor resources are less than the second set of processor resources; and the first set of memory resources are less than the second set of memory resources.

3 . The system of claim 2 , further comprising:
a video camera housing, wherein the video camera housing encloses:
the video image sensor;
the controller; and
a network interface configured to communicate with the network.

4 . The system of claim 1 , wherein:
the image classifier is configured to:
process each video frame in the time-dependent video stream; and
return a binary indicator of the first object type; and
the object detector is configured to:
selectively process, responsive to the image classifier determining that the first object type is present, a subset of video frames to determine the position information for the detected object; and
return position information values for the detected object.

5 . The system of claim 4 , wherein:
the image classifier is further configured to return an image type confidence value; and the object detector is further configured to return an object detected confidence value.

6 . The system of claim 5 , wherein the controller is further configured to:
compare the object detected confidence value to an object verification threshold; responsive to the object detected confidence value meeting the object verification threshold, verify, using the image classifier, the first object type; and responsive to the verification of the first object type being negative, report a detection failure event.

7 . The system of claim 1 , wherein the controller is further configured to:
initiate, responsive to the position information for the detected object, an object tracking algorithm for the detected object to process subsequent video frames of the time-dependent video stream; determine, using the object tracking algorithm, whether the detected object is present in the subsequent video frames of the time-dependent video stream; responsive to the object tracking algorithm determining an object exit event, verify, using the image classifier, the first object type in a video frame corresponding to the object exit event; and responsive to verifying that the first object type is present in the video frame corresponding to the object exit event, report a detection failure event.

8 . The system of claim 1 , wherein:
the image classifier is configured to process the video data from the video image sensor as video frames are received by the controller; and the object detector is configured to selectively process the video data responsive to the image classifier determining that the first object type is present in a classified video data frame.

9 . The system of claim 1 , wherein:
the controller comprises:
a plurality of image classifiers, wherein each image classifier of the plurality of image classifiers is configured for a different object type; and
a plurality of object detectors, wherein each object detector of the plurality of object detectors is configured for a different object type; and
the controller is further configured to:
process the video data through the plurality of image classifiers to determine at least one object type for the first frame;
determine a corresponding object detector from the plurality of object detectors, the corresponding object detector configured to detect an object type corresponding to the at least one object type determined by the plurality of image classifiers; and
process the first video frame using the corresponding object detector to determine the position information for the detected object.

10 . The system of claim 1 , wherein the controller is further configured to send the position information and image data for the detected object for further processing by an analytics engine using a model selected from:
an object recognition model; an object tracking model; and an attribute detection model.

11 . A computer-implemented method, comprising:
receiving video data from a video image sensor, wherein the video data includes a time-dependent video stream of video frames captured by the video image sensor; determining, using an image classifier, whether a first object type is present in a first frame of the video data; determining, using an object detector, position information for a detected object in the video data having the first object type; and sending, over a network, an event notification to a video surveillance application.

12 . The computer-implemented method of claim 11 , further comprising:
configuring a controller to:
use a first set of compute resources for the image classifier; and
use a second set of compute resources for the object detector, wherein the first set of compute resources is less than the second set of compute resources.

13 . The computer-implemented method of claim 12 , wherein:
the controller comprises compute resources including a processor and a memory; the image classifier and the object detector are stored in the memory for execution by the processor; the controller executes:
receiving the video data from the video image sensor;
determining whether the first object type is present;
determining position information for the detected object; and
sending the event notification; and
the controller, the video image sensor, and a network interface for communicating over the network are disposed within a video camera housing.

14 . The computer-implemented method of claim 11 , further comprising:
processing, with the image classifier, each video frame in the video stream; returning, by the image classifier, a binary indicator of the first object type; selectively processing, with the object detector and responsive to the image classifier determining that the first object type is present, a subset of video frames to determine the position information for the detected object; and returning, by the object detector, position information values for the detected object.

15 . The computer-implemented method of claim 14 , further comprising:
returning, by the image classifier, an image type confidence value; and returning, by the object detector, an object detected confidence value.

16 . The computer-implemented method of claim 15 , further comprising:
comparing the object detected confidence value to an object verification threshold; responsive to the object detected confidence value meeting the object verification threshold, verifying, using the image classifier, the first object type; and responsive to the verifica

Assignees

• Western Digital Tech Inc

Inventors

• Xiong Shaomin
• Hirano Toshiki
• Kah Damien
• Nagabhirava Rajeev
• Berman David

Classifications

• G08B13/19608

  Tracking movement of a target, e.g. by detecting an object predefined as a target, using target direction and or velocity to predict its new position · CPC title

• G06T7/70

  Determining position or orientation of objects or cameras (camera calibration G06T7/80) · CPC title

• G06V20/41 Primary

  Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items (segmenting video sequences G06V20/49) · CPC title

• G08B13/19619

  Details of casing · CPC title

• G08B13/19663

  Surveillance related processing done local to the camera · CPC title