Depth and multi-spectral camera

Patent metadata

Field	Value
Publication number	US-10924692-B2
Application number	US-201815974572-A
Country	US
Kind code	B2
Filing date	May 8, 2018
Priority date	May 8, 2018
Publication date	Feb 16, 2021
Grant date	Feb 16, 2021

Abstract

A camera includes a time-of-flight illuminator configured to emit active IR light and a plurality of spectral illuminators, each spectral illuminator configured to emit active spectral light in a different spectral light sub-band. The camera further includes a sensor array that includes a plurality of differential sensors. Each differential sensor is configured to differentially measure both 1) the active IR light, and 2) the active spectral light in each of the different spectral light sub-bands. The camera further includes an output machine operatively connected to the sensor array. The output machine is configured to output a matrix of pixels based on time-multiplexed measurements of the sensor array. Each pixel of the matrix includes 1) a depth value, and 2) a plurality of spectral values. Each of the plurality of spectral values corresponds to a spectral light sub-band of one of the plurality of spectral illuminators.

First claim

The invention claimed is:

1. A camera comprising:
a sensor array including a plurality of differential sensors each configured to measure light over spectral and infrared (IR) bands so as to determine both spectral values and depth values for each of the plurality of differential sensors;
a time-of-flight illuminator configured to emit active IR light in an IR light sub-band;
a plurality of spectral illuminators, each spectral illuminator configured to emit active spectral light in a different spectral light sub-band;
a time-of-flight controller machine configured to:
activate the time-of-flight illuminator to illuminate a subject with the active IR light,
address each of the differential sensors to measure the active IR light emitted from the time-of-flight illuminator and reflected from the subject back to each of the differential sensors, and
determine a depth value for each of the differential sensors based on a time of flight of the active IR light,

a spectral controller machine configured to, for each of the plurality of spectral illuminators:
activate the spectral illuminator to illuminate the subject with active spectral light in the spectral light sub-band of the spectral illuminator,
address each of the differential sensors for a first duration in which the spectral illuminator is activated and address each of the differential sensors for a second duration in which the spectral illuminator is turned off to differentially measure ambient light and the active spectral light emitted from the spectral illuminator in the spectral light sub-band and reflected from the subject back to the differential sensor, and
for each of the plurality of differential sensors, determine a spectral value for the spectral light sub-band based on the depth value and a differential measurement for the differential sensor; and

an output machine configured to output a matrix of pixels, each pixel including the depth value and the spectral value for each spectral light sub-band.

2. The camera of claim 1 , wherein the time-of-flight illuminator is configured to modulate the active IR light, and wherein the time-of-flight controller machine is configured to determine the depth value for each of the differential sensors based on a phase offset of the modulated active IR light reflected from the subject back to the differential sensors.

3. The camera of claim 1 , wherein the time-of-flight controller machine is configured to, for each of the plurality of differential sensors, determine a spectral value for the IR light sub-band based on a measurement of active IR light emitted from the time-of-flight illuminator and reflected from the subject back to the differential sensor, and wherein the output machine is configured to output a spectral value for the IR light sub-band.

4. The camera of claim 1 , wherein the time-of-flight controller machine is configured to:
repeatedly activate the time-of-flight illuminator to repeatedly illuminate the subject with active IR light,
for each activation of the time-of-flight illuminator, address each of the differential sensors for a third duration in which the time-of-flight illuminator is activated and address each of the differential sensors for a fourth duration in which the time-of-flight illuminator is turned off to differentially measure the ambient light and the active IR light emitted from the time-of-flight illuminator and reflected from the subject back to the differential sensors, and
determine the depth value for each of the differential sensors based on the plurality of differential measurements of the ambient light and the active IR light emitted from the time-of-flight illuminator and reflected from the subject back to the differential sensors.

5. The camera of claim 4 , wherein the spectral controller machine is configured to interleave activations of the plurality of spectral illuminators with the activations of the time-of-flight illuminator.

6. The camera of claim 4 , wherein the time-of-flight controller machine is configured to repeatedly activate the time-of-flight illuminator in a first period, and wherein the spectral controller machine is configured to individually activate each of the plurality of spectral illuminators in a second period different than the first period.

7. The camera of claim 1 , wherein the camera includes more than three spectral illuminators.

8. The camera of claim 1 , wherein the time-of-flight illuminator includes an IR laser configured to emit IR light.

9. The camera of claim 1 , wherein each of the plurality of spectral illuminators includes a light emitting diode configured to emit spectral light.

10. The camera of claim 1 , wherein each differential sensor includes a first region and a second region each configured to measure spectral light, wherein the spectral controller machine is configured, for each of the plurality of spectral illuminators, to activate the first region of each of the differential sensors to measure the active spectral light emitted from the spectral illuminator in the spectral light sub-band and reflected from the subject back to the differential sensor and activate the second region of each of the differential sensors to measure the ambient light.

11. A depth and multi-spectral image acquisition method, the method comprising:
activating a time-of-flight illuminator of a camera to illuminate a subject with active IR light in an IR light sub-band;
addressing each of a plurality of differential sensors of a sensor array for a first duration in which the time-of-flight illuminator is activated and address each of the differential sensors for a second duration in which the time-of-flight illuminator is turned off to differentially measure ambient light and the active IR light emitted from the time-of-flight illuminator and reflected from the subject back to each of the plurality of differential sensors;
determining a depth value for each of the plurality of differential sensors based on a time of flight of the active IR light;
for each of a plurality of spectral illuminators of the camera,
activating the spectral illuminator to illuminate the subject with active spectral light in a spectral light sub-band of the spectral illuminator,
addressing each of the plurality of differential sensors for a third duration in which the spectral illuminator is activated and addressing each of the plurality of differential sensors for a fourth duration in which the spectral illuminator is turned off to differentially measure the ambient light and the active spectral light emitted from the spectral illuminator in the spectral light sub-band and reflected from the subject back to the differential sensor,
for each of the plurality of differential sensors, determining a spectral value for the spectral light sub-band based on the depth value and a differential measurement for the differential sensor; and

outputting a matrix of pixels, each pixel including the depth value and the spectral value for each spectral light sub-band.

12. The method of claim 11 , wherein the time-of-flight illuminator is configured to modulate the active IR light, and the depth value for each of the differential sensors is determined based on a phase offset of the modulated active IR light reflected from the subject back to the differential sensors.

13. The method of claim 12 , further comprising:
for each of the plurality of differential sensors, determining a spectral value for the IR light sub-band based on a measurement of active IR light emitted from the time-of-flight illuminator and reflected from the subject back to the differential sensor, and wherein each pixel of the matrix of pixels i

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Ortiz Egea Sergio
• Fenton Michael Scott
• Bamji Cyrus Soli

Classifications

• H04N23/60 Primary

  Control of cameras or camera modules · CPC title

• H04N23/74

  by influencing the scene brightness using illuminating means · CPC title

• H04N23/56

  provided with illuminating means · CPC title

• H04N23/55

  Optical parts specially adapted for electronic image sensors; Mounting thereof · CPC title

• H04N23/54

  Mounting of pick-up tubes, electronic image sensors, deviation or focusing coils · CPC title