System and method for crowd counting and tracking

Patent metadata

Field	Value
Publication number	US-2016110613-A1
Application number	US-201414518981-A
Country	US
Kind code	A1
Filing date	Oct 20, 2014
Priority date	Oct 20, 2014
Publication date	Apr 21, 2016
Grant date	—

Abstract

A video analytic system includes a depth stream sensor, spatial analysis module, temporal analysis module, and analytics module. The spatial analysis module iteratively identifies objects of interest based on local maximum or minimum depth stream values within each frame, removes identified objects of interest, and repeats until all objects of interest have been identified. The temporal analysis module associates each object of interest in the current frame with an object of interest identified in a previous frame, wherein the temporal analysis module utilizes the association between current frame objects of interest and previous frame objects of interest to generate temporal features related to each object of interest. The analytics module detects events based on the received temporal features.

First claim

1 . A video analytic system comprising:
a depth stream sensor that collects depth stream data; a spatial analysis module connected to receive and analyze the depth stream data on a frame by frame basis, wherein the spatial analysis module iteratively identifies objects of interest based on local maximum or minimum depth stream values within each frame, removes identified objects of interest, and repeats until all objects of interest have been identified, wherein the spatial analysis module comprises:
a separator module that applies an adaptive depth threshold to remove those pixels located in a depth range not likely to correspond to objects of interest;
a local minimum depth detector module that identifies within the remaining pixels a minimum depth pixel corresponding to an object located closest to the depth stream sensor; and
a flood fill module that receives the minimum depth pixel from the minimum depth detector and identifies or fills neighboring pixels that should be included as part of the identified object of interest
a temporal analysis module connected to receive objects of interest identified by the spatial analysis module in a frame, wherein the temporal analysis module associates each object of interest in the current frame with an object of interest identified in a previous frame, wherein the temporal analysis module utilizes the association between current frame objects of interest and previous frame objects of interest to generate temporal features related to each object of interest; and an analytics module connected to receive the temporal features generated by the temporal analysis module, wherein the analytics module detects events based on the received temporal features.

2 . (canceled)

3 . The video analytic system of claim 1 , wherein the separator module is a background/foreground separator module that applies a minimum adaptive depth threshold and maximum adaptive depth threshold to remove those pixels associated with background or foreground objects.

4 . (canceled)

5 . The video analytic system of claim 1 , wherein the flood fill module utilizes a threshold fill depth that identifies or fills all pixels located within a minimum depth of the minimum depth associated with the identified object.

6 . The video analytic system of claim 1 , wherein the flood fill module utilizes a rate of change threshold associated with pixel depth to determine those pixels to be included as part of the identified object.

7 . The video analytic system of claim 1 , wherein the flood fill module utilizes a combination of threshold fill depth and rate of change thresholds to identify those pixels to be included as part of the identified object.

8 . The video analytic system of claim 1 , wherein all pixels identified by the flood fill module as part of the identified object are removed from the current frame and stored as a detected object of interest.

9 . The video analytic system of claim 8 , wherein following removal of an identified object form the current frame, the minimum depth detector module detects a subsequent minimum depth pixel and provides the subsequently detected minimum depth pixel to the flood fill module to identify another object of interest, until no additional minimum depth pixels remain.

10 . The video analytic system of claim 1 , wherein the spatial analysis module further includes:
a classifier module that uses one or more features associated with each identified object of interest to determine whether the object of interest has been correctly identified; and a filtering module that receives input from the temporal analysis module regarding objects of interest identified in previous frames to determine whether each object of interest in the current frame has been correctly identified.

11 . The video analytic system of claim 10 , wherein the classifier module compares features associated with each object of interest with expected features, wherein features include one or more of size, shape, and color of the object of interest.

12 . The video analytic system of claim 10 , wherein the filtering module discards an object of interest appearing in the current frame that cannot be associated with an object of interest appearing in a previous frame.

13 . The video analytic system of claim 1 , wherein the temporal analysis module includes:
an association module connected to receive information about each object of interest identified by the spatial analysis module with respect to the current frame, wherein the association module compares current frame objects of interest with previous frame objects of interest and associates a current frame object of interest with a previous frame object of interest; a validation module connected to receive the associations made by the association module, wherein the validation module validates that the current frame object of interest and previous frame object of interest refer to the same object; a reliability module connected to receive the validated associations from the validation module, wherein the reliability module determines a reliability value associated with each detected object of interest based, in part, on a number of frames in which the object has been tracked and associated with previous frame objects of interest; and an output module configured to calculate attributes of each object of interest based on associations between current frame objects of interest and previous frame objects of interest.

14 . The video analytic system of claim 32 , wherein the association module generates an association between a current frame object of interest and a previous frame object of interest based on proximity of the objects of interest to one another.

15 . The video analytic system of claim 14 , wherein the association module generates an association between the current frame object of interest and the previous frame object of interest based, in addition, on attributes associated with each object of interest.

16 . The video analytic system of claim 14 , wherein the association module generates an association between the current frame object of interest and the previous frame object of interest based, in addition, on motion attributes associated with each object of interest including one or more of direction and velocity.

17 . The video analytic system of claim 32 , wherein the validation module utilizes features associated with a current frame object of interest and features associated with a previous frame object of interest to verify that the association is made between the same object, wherein features utilized to verify the association include one or more of size, shape, depth, and color of the current frame object of interest and previous frame object of interest.

18 . The video analytic system of claim 32 , wherein the output module calculates attributes including speed and direction of the object of interest.

19 . The video analytic system of claim 10 , wherein the spatial analysis module is configured to identify as an object of interest a user's head, wherein the minimum depth detector module recognizes a minimum depth value as representing a top of user's head, and wherein the flood fill module identifies neighboring pixels identified as part of the user's head.

20 . The video analytic system of claim 19 , wherein the classifier module validates identification of objects of interest as a user's head based on one or more features associated with each object as compared with expected values, wherein features include one or more of

Assignees

• Univ King Abdullah Sci & Tech

Inventors

• Ghanem Bernard
• Schneider Jens
• Shalaby Mohamed
• Elnily Usama

Classifications

• G06V20/53

  Recognition of crowd images, e.g. recognition of crowd congestion · CPC title

• G06T7/11 Primary

  Region-based segmentation · CPC title

• G06T2207/30242

  Counting objects in image · CPC title

• G06K9/00778 Primary

  Physics · mapped topic

• G06T7/0065

  Physics · mapped topic