System and method for video content analysis using depth sensing

Patent metadata

Field	Value
Publication number	US-9247211-B2
Application number	US-201313744254-A
Country	US
Kind code	B2
Filing date	Jan 17, 2013
Priority date	Jan 17, 2012
Publication date	Jan 26, 2016
Grant date	Jan 26, 2016

Abstract

A method and system for performing video content analysis based on two-dimensional image data and depth data are disclosed. Video content analysis may be performed on the two-dimensional image data, and then the depth data may be used along with the results of the video content analysis of the two-dimensional data for tracking and event detection.

First claim

What is claimed is:

1. A video content analysis method comprising:
receiving a video sequence that includes a plurality of frames, each frame including a video image;
for each frame, receiving two-dimensional (2D) image data and also receiving depth data;
processing the 2D image data of the video sequence to differentiate foreground data from background data and to detect one or more blobs comprised of the foreground data, the one or more blobs corresponding to one or more objects, wherein differentiating the foreground data from the background data is performed without analyzing the depth data;
for each detected blob, using the depth data to determine whether at least part of the blob corresponds to at least part of a target by at least (1) mapping the blob to a set of Z-planes; (2) determining that on at least some Z-planes the blob is clustered into different blob regions corresponding to two objects separated in space; and (3) grouping the separated blob regions of the Z-planes into two physical objects by checking their spatial overlaps, wherein one of the physical objects corresponds to the target; and
after it is determined that at least part of a blob corresponds to at least part of a target, tracking the target and detecting at least one event associated with the target.

2. The method of claim 1 , wherein using the depth data to determine whether at least part of each blob corresponds to at least part of a target includes: using the depth data to determine that only part of a first blob corresponds to a first target.

3. The method of claim 2 , wherein using the depth data to determine whether at least part of each blob corresponds to at least part of a target includes:
using the depth data to determine that part of the first blob does not correspond to the first target.

4. The method of claim 2 , wherein using the depth data to determine whether at least part of each blob corresponds to at least part of a target includes:
using the depth data to determine that part of the first blob corresponds to a second target different from the first target.

5. The method of claim 4 , wherein a first part of the first blob corresponds to the first target and a second part of the first blob corresponds to a second target, one of the first and second target occluding at least part of the other.

6. The method of claim 2 , wherein using the depth data to determine whether at least part of each blob corresponds to at least part of a target includes:
using the depth data to determine that a second blob combined with part or all of the first blob correspond to a second target.

7. The method of claim 1 , wherein the 2D image data includes RGB data for each pixel in the video image.

8. The method of claim 7 , wherein only pixels of foreground data are projected onto the set of Z-planes.

9. The method of claim 1 , wherein determining whether at least part of each blob corresponds to at least part of a target is performed without analyzing depth data associated with the background data.

10. The method of claim 1 , wherein using the depth data to determine whether at least part of each blob corresponds to at least part of a target comprises:
using the depth data to determine one or more of a height and a volume of the blob; and
using one or more of the height and the volume of the blob to determine whether at least part of the blob corresponds to a target.

11. The method of claim 10 , wherein determining whether the blob corresponds to a target includes determining whether the blob is a person.

12. A video content analysis method comprising:
receiving a video sequence that includes a plurality of frames, each frame including a video image;
for each frame, receiving two-dimensional (2D) image data and also receiving depth data;
processing the 2D image data of the video sequence to differentiate foreground data from background data and to detect one or more blobs comprised of the foreground data, the one or more blobs corresponding to one or more objects, wherein differentiating the foreground data from the background data is performed without analyzing the depth data;
for each detected blob, using the depth data to determine whether to track at least a first part of the blob as a target; and
after it is determined to track the target, detecting at least one event associated with the target,
wherein determining whether to track at least the first part of the blob as a target includes:
mapping the blob to a set of Z-planes;
determining that on at least some Z-planes the blob is clustered into different blob regions corresponding to the first part of the blob and a second part of the blob separated in space; and
grouping blob slices corresponding to the Z-planes from the first part of the blob to correspond to a physical object by checking their spatial overlaps, wherein the physical object corresponds to the target.

13. The method of claim 12 , wherein using the depth data to determine whether to track at least part of the blob as a target includes:
determining that part of a first blob corresponds to a first target.

14. The method of claim 13 , wherein using the depth data to determine whether to track at least part of the blob as a target includes:
determining that part of the first blob does not correspond to the first target.

15. The method of claim 13 , wherein using the depth data to determine whether to track at least part of the blob as a target includes:
determining that a part of the first blob corresponds to a second target different from the first target.

16. The method of claim 15 , wherein determining that part of the first blob corresponds to the second target includes determining that the first blob corresponds to a first person and the second blob corresponds to a second person, one of the first and second person occluding at least part of the other.

17. The method of claim 13 , wherein using the depth data to determine whether to track at least part of the blob as a target includes:
determining that a second blob combined with part or all of the first blob corresponds to a second target.

18. The method of claim 12 , wherein the 2D image data includes RGB data for each pixel in the video image.

19. The method of claim 18 , wherein only pixels of foreground data are projected onto the set of Z-planes.

20. The method of claim 19 , wherein determining whether to track at least part of the blob as a target is performed without analyzing depth data associated with the background data.

21. The method of claim 12 , wherein using the depth data to determine whether to track at least part of the blob as a target includes:
using the depth data to determine one or more of a height and a volume of the blob; and
using one or more of the height and volume of the blob to determine whether the blob corresponds to a target.

22. The method of claim 21 , wherein determining whether the blob corresponds to a target includes determining whether the blob is a person.

Assignees

• Avigilon Fortress Corp

Inventors

• Zhang Zhong
• Myers Gary W
• Venetianer Peter L

Classifications

• A61B5/1072 Primary

  measuring distances on the body, e.g. measuring length, height or thickness (A61B5/1076 takes precedence) · CPC title

• A61B5/0077

  Devices for viewing the surface of the body, e.g. camera, magnifying lens · CPC title

• A61B5/1079

  using optical or photographic means · CPC title

• A61B5/1113

  Local tracking of patients, e.g. in a hospital or private home · CPC title

• G06K9/00369

  Physics · mapped topic