Using a depth map of a monitored scene to identify floors, walls, and ceilings

Patent metadata

Field	Value
Publication number	US-9554064-B2
Application number	US-201514745259-A
Country	US
Kind code	B2
Filing date	Jun 19, 2015
Priority date	Jun 12, 2015
Publication date	Jan 24, 2017
Grant date	Jan 24, 2017

Abstract

A camera system includes memory, a lens assembly to direct light from a scene onto an image sensing element, an image sensing element configured to receive light from the scene via the lens assembly, at least one infrared illuminator configured to transmit infrared light, and a processor, coupled to the image sensing element and the at least one infrared illuminator. The processor is configured to operate the illuminators and the image sensing element in a first mode whereby infrared light transmitted by the illuminators and reflected from the scene is used to generate a two-dimensional image of the scene. The processor is also configured to operate the illuminators and the image sensing element in a second mode whereby infrared light transmitted by the illuminators and reflected from the scene is used to identify a planar surface in the scene.

First claim

What is claimed is:

1. A camera device, comprising:
a memory configured to store information;
a lens assembly arranged to direct light from a scene;
a plurality of image sensing elements configured to receive light from the scene via the lens assembly;
a plurality of infrared illuminators configured to transmit infrared light; and
a processor, coupled to the plurality of image sensing elements and the plurality of infrared illuminators and configured to:
operate the plurality of infrared illuminators and the image sensing element in a first mode whereby infrared light transmitted by the plurality of infrared illuminators and reflected from the scene is used to generate a two-dimensional image of the scene; and
operate the plurality of infrared illuminators and the image sensing element in a second mode, including:
sequentially activating a plurality of distinct subsets of the illuminators, each subset illuminating a different portion of the scene;
for each subset of illuminators, measuring an intensity of reflected light from the scene for each image sensing element of the plurality of image sensing elements;
for each image sensing element:
generating a light intensity vector based on the measured intensities for the element, wherein the light intensity vector has a component corresponding to each subset of illuminators; and
estimating a depth in the scene using the light intensity vector and a lookup table; and

identifying a planar surface in the scene using the estimated depths.

2. The camera device of claim 1 , wherein the first mode is operated at night.

3. The camera device of claim 1 , wherein in the first mode all of the infrared illuminators are activated to generate the two-dimensional image.

4. The camera device of claim 1 , wherein the second mode is a three-dimensional surface detection mode.

5. The camera device of claim 1 , wherein each of the subsets of illuminators is at a different angle relative to the plurality of image sensing elements.

6. The camera device of claim 1 , wherein the processor is further configured to: aggregate the estimated pixel depths into a data structure for identifying planar surfaces in the scene.

7. The camera device of claim 6 , wherein identifying planar surfaces in the scene using the estimated pixel depths comprises transmitting the data structure to a planar surface detection module configured to identify planar surfaces using a depth map constructed from the data structure.

8. The camera device of claim 6 , wherein identifying planar surfaces in the scene using the estimated pixel depths comprises:
using the data structure to identify a region in the scene; and
fitting a plane to a set of points in the region.

9. The camera device of claim 8 , wherein the processor is further configured to:
determine that the region corresponds to a planar surface in the scene when a plane fitting residual error for the fitted plane is less than a predefined value.

10. The camera device of claim 6 , wherein the processor is further configured to:
construct a depth map of the scene using the data structure;
use the depth map to compute a binary depth edge map for the scene, wherein the binary depth edge map identifies which points in the depth map comprise depth discontinuities; and
select a contiguous region from the binary depth edge map.

11. The camera device of claim 8 , wherein the processor is further configured to determine an orientation of the fitted plane.

12. The camera device of claim 11 , wherein the orientation of the plane is upwards, and the fitted plane is determined to be a floor.

13. The camera device of claim 11 , wherein the orientation of the plane is downwards, and the fitted plane is determined to be a ceiling.

14. The camera device of claim 11 , wherein the orientation of the plane is horizontal, and the fitted plane is determined to be a wall.

15. A method, comprising:
at a camera device having a memory configured to store information, a lens assembly arranged to direct light from a scene, a plurality of image sensing elements configured to receive light from the scene via the lens assembly, a plurality of infrared illuminators configured to transmit infrared light, and a processor, coupled to the plurality of image sensing elements and the plurality of infrared illuminators:
operating the plurality of infrared illuminators and the image sensing element in a first mode whereby infrared light transmitted by the plurality of infrared illuminators and reflected from the scene is used to generate a two-dimensional image of the scene; and
operating the plurality of infrared illuminators and the image sensing element in a second mode, including:
sequentially activating a plurality of distinct subsets of the illuminators, each subset illuminating a different portion of the scene;
for each subset of illuminators, measuring an intensity of reflected light from the scene for each pixel of a plurality of pixels, wherein each pixel corresponds to one or more image sensing elements of the plurality of image sensing elements;
for each pixel:
generating a light intensity vector based on the measured intensities for the pixel, wherein the light intensity vector has a component corresponding to each subset of illuminators; and
estimating a pixel depth in the scene using the light intensity vector and a lookup table; and

identifying a planar surface in the scene using the estimated pixel depths.

16. The method of claim 15 , wherein the first mode is configured to operate at night.

17. The method of claim 15 , further comprising aggregating the estimated pixel depths into a data structure for identifying planar surfaces in the scene.

18. A non-transitory computer-readable storage medium storing one or more programs configured for execution by a camera device having a memory configured to store information, a lens assembly arranged to direct light from a scene, plurality of image sensing elements configured to receive light from the scene via the lens assembly, a plurality of infrared illuminators configured to transmit infrared light, and a processor, coupled to the plurality of image sensing elements and the plurality of infrared illuminators, wherein the one or more programs comprise instructions for:
operating the plurality of infrared illuminators and the image sensing element in a first mode whereby infrared light transmitted by the plurality of infrared illuminators and reflected from the scene is used to generate a two-dimensional image of the scene; and
operating the plurality of infrared illuminators and the image sensing element in a second mode, including:
sequentially activating a plurality of distinct subsets of the illuminators, each subset illuminating a different portion of the scene;
for each subset of illuminators, measuring an intensity of reflected light from the scene for each pixel of a plurality of pixels, wherein each pixel corresponds to one or more image sensing elements of the plurality of image sensing elements;
for each pixel:
generating a light intensity vector based on the measured intensities for the pixel, wherein the light intensity vector has a component corresponding to each subset of illuminators; and
estimating a pixel depth in the scene using the light intensity vector and a lookup table; and

identifying a planar surface in the scene using the estimated pixel depths.

19. The non-transitory computer-readable storage medium of claim 18 , wherein the first mode is configured to operate at night.

Assignees

• Google Inc

Inventors

• Dixon Michael
• Shin Dongeek

Classifications

• G06V10/762

  using clustering, e.g. of similar faces in social networks · CPC title

• H04N7/188 Primary

  Capturing isolated or intermittent images triggered by the occurrence of a predetermined event, e.g. an object reaching a predetermined position (signal generation from motion picture films H04N5/253) · CPC title

• H04N23/56

  provided with illuminating means · CPC title

• H04N23/74

  by influencing the scene brightness using illuminating means · CPC title

• G06F18/23

  Clustering techniques · CPC title