System and method for 3D reconstruction using multiple multi-channel cameras

Patent metadata

Field	Value
Publication number	US-9392262-B2
Application number	US-201514641092-A
Country	US
Kind code	B2
Filing date	Mar 6, 2015
Priority date	Mar 7, 2014
Publication date	Jul 12, 2016
Grant date	Jul 12, 2016

Abstract

A method for three-dimensional reconstruction of a scene includes: controlling a projection source to emit invisible light toward the scene; while the projection source is emitting light: controlling a first multi-channel image sensor to capture a first image, the first multi-channel image sensor including visible light detecting pixels and invisible light detecting pixels, the first image including a first invisible light channel and a first visible light channel; controlling a second multi-channel image sensor to capture a second image substantially simultaneously as the capture of the first image, the second multi-channel image sensor including visible light detecting pixels and invisible light detecting pixels, the second image including a second invisible light channel and a second visible light channel; performing stereo analysis of the first image and the second image in accordance with the invisible light channels and the visible light channels to generate a depth map.

First claim

What is claimed is:

1. A system for three-dimensional reconstruction of a scene, the system comprising:
a first camera comprising a first lens and a first multi-channel image sensor comprising a plurality of visible light detecting pixels and a plurality of invisible light detecting pixels;
a second camera comprising a second lens and a second multi-channel image sensor comprising a plurality of visible light detecting pixels and a plurality of invisible light detecting pixels;
a projection source;
a processor; and
a memory storing instructions that, when executed by the processor, cause the processor to:
control the projection source to emit invisible light toward the scene;
control the first multi-channel image sensor to capture a first image, the first image comprising a first invisible light channel and a first visible light channel;
control the second multi-channel image sensor to capture a second image substantially simultaneously as the capture of the first image by the first multi-channel image sensor, the second image comprising a second invisible light channel and a second visible light channel;
perform stereo analysis of the first image and the second image in accordance with the first and second invisible light channels and the first and second visible light channels to generate a depth map by:
performing stereo analysis of the first invisible light channel and the second invisible light channel to generate a coarse depth map; and
applying a cross-bilateral filter on the coarse depth map with weights from the first visible light channel and the second visible light channel to generate the depth map; and

output the depth map.

2. The system of claim 1 , wherein the memory further stores instructions that, when executed by the processor, cause the processor to perform stereo analysis of the first image and the second image by:
up-sampling the coarse depth map to a resolution of the first and second visible light channels.

3. The system of claim 1 , wherein the projection source is configured to emit infrared light, and
wherein the invisible light detecting pixels are configured to detect infrared light.

4. The system of claim 1 , wherein the first visible light channel comprises red, blue, and green color channels, and
wherein the second visible light channel comprises red, blue, and green color channels.

5. The system of claim 1 , wherein the projection source is configured to emit the invisible light in a pattern.

6. A system for three-dimensional reconstruction of a scene, the system comprising:
a first camera comprising a first lens and a first multi-channel image sensor comprising a plurality of visible light detecting pixels and a plurality of invisible light detecting pixels;
a second camera comprising a second lens and a second multi-channel image sensor comprising a plurality of visible light detecting pixels and a plurality of invisible light detecting pixels;
a projection source;
a processor; and
a memory storing instructions that, when executed by the processor, cause the processor to:
control the first multi-channel image sensor to capture an initial image before controlling the projection source to emit invisible light toward the scene;
identify a region of interest in the initial image;
control the projection source to emit invisible light toward the scene;
control the first multi-channel image sensor to capture a first image, the first image comprising a first invisible light channel and a first visible light channel;
control the second multi-channel image sensor to capture a second image substantially simultaneously as the capture of the first image by the first multi-channel image sensor, the second image comprising a second invisible light channel and a second visible light channel;
perform stereo analysis of the first image and the second image in accordance with the first and second invisible light channels and the first and second visible light channels to generate a depth map; and
output the depth map.

7. The system of claim 6 , wherein the memory further stores instructions that, when executed by the processor, cause the processor to:
detect a feature within the region of interest;
identify a minimum disparity bound in accordance with the feature; and
perform the stereo analysis in accordance with the minimum disparity bound and only within the region of interest.

8. The system of claim 7 , wherein the memory further stores instructions that, when executed by the processor, cause the processor to:
identify a maximum disparity bound in accordance with the feature; and
perform the stereo analysis in accordance with the maximum disparity bound.

9. The system of claim 7 , wherein the projection source comprises a scanner mirror and a light source configured to emit the invisible light, and
wherein the processor is configured to control the scanner mirror to direct the invisible light emitted by the light source toward only a portion of the scene corresponding to the region of interest.

10. A system for three-dimensional reconstruction of a scene, the system comprising:
a first camera comprising a first lens and a first multi-channel image sensor comprising a plurality of visible light detecting pixels and a plurality of invisible light detecting pixels;
a second camera comprising a second lens and a second multi-channel image sensor comprising a plurality of visible light detecting pixels and a plurality of invisible light detecting pixels;
a projection source;
a processor; and
a memory storing instructions that, when executed by the processor, cause the processor to:
control the projection source to emit invisible light toward the scene;
control the first multi-channel image sensor to capture a first image, the first image comprising a first invisible light channel and a first visible light channel;
control the second multi-channel image sensor to capture a second image substantially simultaneously as the capture of the first image by the first multi-channel image sensor, the second image comprising a second invisible light channel and a second visible light channel;
perform stereo analysis of the first image and the second image in accordance with the first and second invisible light channels and the first and second visible light channels to generate a depth map by:
performing stereo analysis of the first and second invisible light channels to generate an invisible light depth map;
performing stereo analysis of the first and second visible light channels to generate a visible light depth map; and
combining the invisible light depth map and the visible light depth map to generate the depth map; and

output the depth map.

11. A method for three-dimensional reconstruction of a scene, the method comprising:
controlling, by a processor, a projection source to emit invisible light toward the scene:
controlling a first multi-channel image sensor to capture a first image, the first multi-channel image sensor comprising a first camera comprising a first lens and a plurality of visible light detecting pixels and a plurality of invisible light detecting pixels, the first image comprising a first invisible light channel and a first visible light channel;
controlling a second multi-channel image sensor to capture a second image substantially simultaneously as the capture of the first image by the first multi-channel image sensor, the second multi-channel image sensor comprising a second camera comprising a second lens and a plurality of visible light detecting pixels and a plurality of invisible light detecting pixels, the second image comprising a second invisible light channel and a second visible light

Assignees

• Aquifi Inc

Inventors

• Dal Mutto Carlo
• Rafii Abbas
• Demirdjian David

Classifications

• H04N25/134

  based on three different wavelength filter elements · CPC title

• H04N25/135

  based on four or more different wavelength filter elements · CPC title

• H04N23/61

  based on recognised objects · CPC title

• H04N25/131

  including elements passing infrared wavelengths · CPC title

• H04N23/11

  for generating image signals from visible and infrared light wavelengths · CPC title