System and method for building automation using video content analysis with depth sensing

Patent metadata

Field	Value
Publication number	US-9530060-B2
Application number	US-201313744264-A
Country	US
Kind code	B2
Filing date	Jan 17, 2013
Priority date	Jan 17, 2012
Publication date	Dec 27, 2016
Grant date	Dec 27, 2016

Abstract

A method and system for monitoring buildings (including houses and office buildings) by performing video content analysis based on two-dimensional image data and depth data are disclosed. Occupation and use of such buildings may be monitored with higher accuracy to provide higher energy efficiency usage, to control operation of components therein, and/or provide better security. Height data may be obtained from depth data to provide greater reliability in object detection, object classification and/or event detection.

First claim

What is claimed is:

1. A method of monitoring a building comprising:
taking a video within a location in the building with a video sensor, the video comprising a plurality of frames, each frame including 2D image data;
for each frame, receiving depth data associated with the 2D image data, the depth data corresponding to one or more distances from the video sensor to features represented by the 2D image data;
analyzing the 2D image data including performing 2D analysis on the 2D image data to detect one or more image blobs in the video;
using the depth data, projecting an image blob of the one or more image blobs onto a plurality of Z-planes, thereby creating a plurality of blob slices;
separating the Z-planes into a ground plane and non-round planes based on a height threshold, so that at least a first blob slice on the ground plane is at or below the height threshold, and a second blob slice and additional blob slices on the non-ground planes are above the height threshold;
creating a refined blob that includes the second blob slice, the additional blob slices, and only a portion of the first blob slice;
performing object detection on the refined blob, to determine that the blob corresponds to a human object, thereby detecting a person in the video;
based on the detected person and one or more additional detected people, counting a number of people at the location in the building; and
controlling a system of the building in response to the number of people counted.

2. The method of claim 1 , wherein
controlling a system comprises at least one of turning on lights and turning off lights of the building.

3. The method of claim 1 , wherein
controlling a system comprises setting a thermostat temperature of a heating system or a cooling system of the building.

4. The method of claim 1 , further comprising providing an alert that at least one of a room or area is utilized when a number of people counted exceeds a predetermined value.

5. The method of claim 4 , comprising providing an alert that a conference room of a building is utilized.

6. The method of claim 1 , comprising counting a number of people that are sitting.

7. The method of claim 6 , comprising detecting that people are sitting in front of a television, and reducing light usage of the building in response to detecting that people are sitting in front of a television.

8. The method of claim 1 , further comprising detecting that a person has gone to bed.

9. The method of claim 8 , further comprising detecting that a person has gone to bed by determining the person has lied down at a location other than the floor.

10. The method of claim 8 , further comprising detecting that a person has gone to bed by detecting a height of the person.

11. The method of claim 8 , further comprising reducing light usage of the building in response to detecting that the person has gone to bed.

12. The method of claim 8 , further comprising controlling window blinds in response to detecting that the person has gone to bed.

13. The method of claim 1 , further comprising detecting that a person has gotten out of bed.

14. The method of claim 13 , further comprising, in response to detecting that a persona has gotten out of bed, controlling at least one of:
lights of the building;
window blinds;
heating of the building; and
cooling of the building.

15. The method of claim 1 , further comprising:
determining a number of people within an area near elevators by the analyzing the image data and the depth data;
controlling elevator operation in response to the counting of the number of people.

16. The method of claim 15 ,
wherein the depth data is used to determine heights of detected objects; and
the determined heights of the detected objects are used to determine if the detected objects are people.

17. The method of claim 1 , further comprising:
determining a number of people within an elevator by the analyzing the image data and the depth data;
controlling operation of the elevator in response to the counting of the number of people.

18. The method of claim 17 ,
wherein the depth data is used to determine heights of detected objects; and
the determined heights of the detected objects is are used to determine if the detected objects are people.

19. The method of claim 1 , further comprising:
determining an action of a detected person by analyzing depth data associated with the person;
controlling a device within the building in response to the detected action.

20. The method of claim 19 , wherein determining an action of the detected person comprises detecting that the person is waving an arm.

21. The method of claim 20 , wherein controlling a device within the building in response to the detected action comprises controlling at least one of:
light operation;
projector operation; and
blind operation.

22. The method of claim 1 , further comprising:
creating the refined blob by:
determining a first region of the first blob slice that overlaps the second blob slice from a top down view; and
including the first region of the first blob slice in the refined blob, without including a second region of the first blob slice in the refined blob.

23. The method of claim 22 , wherein the second region of the first blob slice corresponds to a shadow or reflection.

24. A method of monitoring a building comprising:
taking a video within a location in the building, the video comprising a plurality of frames, each frame including 2D image data;
for each frame, receiving depth data associated with the 2D image data, the depth data corresponding to one or more distances from a depth sensor to features represented by the 2D image data;
analyzing the 2D image data to detect at least a first image blob in the video;
using the depth data, projecting the image blob onto a plurality of Z-planes, thereby creating a plurality of blob slices;
based on a height threshold, separating the blob slices into a ground plane blob slice, and non-ground plane blob slices;
creating a refined blob that includes non-ground plane blob slices, and only a portion of the ground plane blob slice;
performing object detection on the refined blob, to determine that the blob corresponds to a human object, thereby detecting a person in the video; and
controlling a system of the building in response to the detected person.

25. The method of claim 24 , wherein the ground plane blob slice corresponds to a shadow or reflection.

26. The method of claim 24 , further comprising:
creating the refined blob by:
determining a first region of the ground plane blob slice that overlaps the non-ground plane blob slices from a top down view; and
including the first region of the ground plane blob slice in the refined blob, without including a second region of the ground plane blob slice in the refined blob.

Assignees

• Avigilon Fortress Corp

Inventors

• Zhang Zhong
• Myers Gary W
• Venetianer Peter L

Classifications

• G06K2009/00738

  Physics · mapped topic

• G06T7/0071

  Physics · mapped topic

• A61B5/1073

  Measuring volume, e.g. of limbs · CPC title

• A61B5/0013

  Medical image data (A61B1/00011, A61B6/56, A61B8/56 take precedence) · CPC title

• A61B5/1079

  using optical or photographic means · CPC title