Using a depth map of a monitored scene to identify floors, walls, and ceilings

Patent metadata

Field	Value
Publication number	US-9613423-B2
Application number	US-201514738817-A
Country	US
Kind code	B2
Filing date	Jun 12, 2015
Priority date	Jun 12, 2015
Publication date	Apr 4, 2017
Grant date	Apr 4, 2017

Abstract

A process identifies large planar surfaces in scenes. The process receives captured IR images of a scene taken by a 2-dimensional array of image sensors of a camera system. Each IR image is captured when a distinct subset of IR illuminators of the camera system is illuminated. The process constructs a depth map of a scene using IR images and uses the depth map to compute a binary depth edge map for the scene. The binary depth edge map identifies which points in the depth map comprise depth discontinuities. The process identifies contiguous components based on the binary depth edge map and determines that a first component of the contiguous components represents a large planar surface in the scene by: fitting a plane to points in the first component; determining the orientation of the plane; and determining that the plane fitting residual error is less than a predefined threshold.

First claim

What is claimed is:

1. A method of identifying large planar surfaces in scenes, comprising:
at a computing device having one or more processors, and memory storing one or more programs configured for execution by the one or more processors:
receiving a plurality of captured IR images of a scene taken by a 2-dimensional array of image sensors of a camera that has a plurality of IR illuminators proximate to the array of image sensors, wherein each IR image is captured when a distinct subset of the IR illuminators of the camera is illuminated;
constructing a depth map of a scene using the plurality of IR images;
using the depth map to compute a binary depth edge map for the scene, wherein the binary depth edge map specifies lines of depth discontinuities in the depth map;
identifying a plurality of image regions enclosed by lines of depth discontinuities in the binary depth edge map;
determining that a first image region of the plurality of image regions represents a large planar surface in the scene by:
fitting a plane to points in the first component;
determining the orientation of the plane; and
determining that the plane fitting residual error is less than a predefined threshold; and

identifying a region in a displayed RGB image of the scene taken by the camera, wherein the region in the RGB image corresponds to the first image region.

2. The method of claim 1 , wherein the orientation of the plane is upwards, and the plane is determined to be a floor.

3. The method of claim 1 , wherein the orientation of the plane is downwards, and the plane is determined to be a ceiling.

4. The method of claim 1 , wherein the orientation of the plane is horizontal, and the plane is determined to be a wall.

5. The method of claim 1 , wherein the computing device is a server distinct from the camera system.

6. The method of claim 1 , wherein the computing device is included in the camera system.

7. The method of claim 1 , wherein the image sensors are partitioned into a plurality of pixels and wherein constructing the depth map comprises:
for each pixel of the plurality of pixels:
forming a respective vector comprising measured IR light intensity at the respective pixel for each of the plurality of captured IR images; and
estimating a depth in the first scene at the respective pixel using the respective vector and a respective lookup table.

8. The method of claim 1 , further comprising monitoring a video stream provided by the camera to identify motion events, excluding the region in the RGB image corresponding to the first image region, and generating a motion alert when there is detected motion in the scene outside of the region in the RGB image corresponding to the first image region.

9. The method of claim 2 , further comprising estimating a height and tilt angle of the camera relative to the determined floor.

10. A computing device, comprising:
one or more processors;
memory; and
one or more programs stored in the memory configured for execution by the one or more processors, the one or more programs comprising instructions for:
receiving a plurality of captured IR images of a scene taken by a 2-dimensional array of image sensors of a camera that has a plurality of IR illuminators proximate to the array of image sensors, wherein each IR image is captured when a distinct subset of the IR illuminators of the camera is illuminated;
constructing a depth map of a scene using the plurality of IR images;
using the depth map to compute a binary depth edge map for the scene, wherein the binary depth edge map specifies lines of depth discontinuities in the depth map;
identifying a plurality of image regions enclosed by lines of depth discontinuities in the binary depth edge map;
determining that a first image region of the plurality of image regions represents a large planar surface in the scene by:
fitting a plane to points in the first component;
determining the orientation of the plane; and
determining that the plane fitting residual error is less than a predefined threshold; and

identifying a region in a displayed RGB image of the scene taken by the camera, wherein the region in the RGB image corresponds to the first image region.

11. The computing device of claim 10 , wherein the orientation of the plane is upwards, and the plane is determined to be a floor.

12. The computing device of claim 10 , wherein the orientation of the plane is downwards, and the plane is determined to be a ceiling.

13. The computing device of claim 10 , wherein the orientation of the plane is horizontal, and the plane is determined to be a wall.

14. The computing device of claim 10 , wherein the computing device is a server distinct from the camera system.

15. The computing device of claim 10 , wherein the computing device is included in the camera system.

16. The computing device of claim 10 , wherein the image sensors are partitioned into a plurality of pixels and wherein the instructions for constructing the depth map comprise instructions for:
for each pixel of the plurality of pixels:
forming a respective vector comprising measured IR light intensity at the respective pixel for each of the plurality of captured IR images; and
estimating a depth in the first scene at the respective pixel using the respective vector and a respective lookup table.

17. A non-transitory computer readable storage medium storing one or more programs configured for execution by a computing device having one or more processors and memory, the one or more programs comprising instructions for:
receiving a plurality of captured IR images of a scene taken by a 2-dimensional array of image sensors of a camera that has a plurality of IR illuminators proximate to the array of image sensors, wherein each IR image is captured when a distinct subset of the IR illuminators of the camera is illuminated;
constructing a depth map of a scene using the plurality of IR images;
using the depth map to compute a binary depth edge map for the scene, wherein the binary depth edge map specifies lines of depth discontinuities in the depth map;
identifying a plurality of image regions enclosed by lines of depth discontinuities in the binary depth edge map;
determining that a first image region of the plurality of image regions represents a large planar surface in the scene by:
fitting a plane to points in the first component;
determining the orientation of the plane; and
determining that the plane fitting residual error is less than a predefined threshold; and

identifying a region in a displayed RGB image of the scene taken by the camera, wherein the region in the RGB image corresponds to the first image region.

18. The computer readable storage medium of claim 17 , wherein the orientation of the plane is upwards, and the plane is determined to be a floor.

19. The computer readable storage medium of claim 17 , wherein the orientation of the plane is downwards, and the plane is determined to be a ceiling.

20. The computer readable storage medium of claim 17 , wherein the computing device is a server distinct from the camera system.

21. The computer readable storage medium of claim 17 , wherein the computing device is included in the camera system.

22. The computer readable storage medium of claim 17 , wherein the image sensors are partitioned into a plurality of pixels and wherein the instructions for constructing the depth map comprise instructions for:
for each pixel of the plurality of pixels:
forming a

Assignees

• Google Inc

Inventors

• Dixon Michael
• Shin Dongeek

Classifications

• G06V10/762

  using clustering, e.g. of similar faces in social networks · CPC title

• H04N7/188 Primary

  Capturing isolated or intermittent images triggered by the occurrence of a predetermined event, e.g. an object reaching a predetermined position (signal generation from motion picture films H04N5/253) · CPC title

• H04N23/74

  by influencing the scene brightness using illuminating means · CPC title

• G06F18/23

  Clustering techniques · CPC title

• H04N23/56

  provided with illuminating means · CPC title