Methods and apparatus for demultiplexing illumination

Patent metadata

Field	Value
Publication number	US-9451141-B2
Application number	US-201514690159-A
Country	US
Kind code	B2
Filing date	Apr 17, 2015
Priority date	Apr 19, 2014
Publication date	Sep 20, 2016
Grant date	Sep 20, 2016

Abstract

In illustrative implementations of this invention, an imaging system includes multiple light sources that illuminate a scene, and also includes a lock-in time of flight camera. While the scene is illuminated by these light sources, each of the light sources is amplitude-modulated by a different modulation pattern, and a reference signal is applied to the lock-in time-of-flight camera. The modulation patterns and the reference signal are carefully chosen such that the imaging system is able to disentangle, in real time, the respective contributions of the different light sources, and to compute, in real-time, depth of the scene. In some cases, the modulation signals for the light sources are orthogonal to each other and the reference signal is broadband. In some cases, the modulation codes for the light sources and the reference code are optimal codes that are determined by an optimization algorithm.

First claim

What is claimed is:

1. A method comprising, in combination:
(a) a set of multiple light sources illuminating a scene;
(b) a signal generator generating a reference signal and a set of multiple modulation signals, such that, while the light sources illuminate the scene
(i) the modulation signals control amplitude of light emitted by the light sources, causing the amplitude of light emitted by each light source to vary over time,
(ii) the modulation signal for each light source is different than the modulation signal for the other light sources, and
(iii) the modulation signals overlap in time and cause two or more of the light sources to simultaneously emit light during multiple time intervals of the modulation signals;

(c) each pixel in the camera
(i) measuring an incident intensity, which incident intensity is intensity of light that reflects from the scene and is incident on the pixel, and
(ii) outputting data indicative of a cross-correlation of the incident intensity and the reference signal; and

(d) a computer taking the data as an input and estimating, for each pixel of the camera
(i) a separate intensity for each respective light source, out of the set of light sources, which separate intensity is intensity of light that (A) is emitted by the respective light source and not by other light sources, (B) then reflects from the scene and (C) then travels to the pixel, and
(ii) depth of a point in the scene that reflects light to the pixel.

2. The method of claim 1 , wherein the modulation signals are orthogonal to each other.

3. The method of claim 1 , wherein:
(a) the camera has a sampling rate, which sampling rate is equal to the number of raw frames of the camera per second; and
(b) the reference signal is broadband with respect to the sampling rate.

4. The method of claim 1 , wherein:
(a) the camera has a rate sampling rate, which sampling rate is equal to the number of raw frames of the camera per second; and
(b) the modulation signals are orthogonal to each other and the reference signal is broadband with respect to the sampling rate.

5. The method of claim 1 , wherein the camera comprises a lock-in time of flight camera.

6. The method of claim 1 , wherein:
(a) the modulation signals are periodic; and
(b) multiple periods of each of the modulation signals occur during each of the camera.

7. The method of claim 1 , wherein data indicative of the modulation signals and reference signal:
(a) is stored in an electronic memory device; and
(b) is read from the memory device and is used by a signal generator that generates the modulation signals and reference signal.

8. The method of claim 7 , wherein:
(a) before the data is stored in the electronic memory device, a computer performs an algorithm to calculate the modulation signals and the reference signal; and
(b) the algorithm includes calculating a best linear unbiased estimator that solves
{circumflex over (x)} BLUE =( L T T T Σ −1 TL ) −1 L T T T Σ −1 c ω

where (i) c ω =TLx+η and c ω ε m×1 (ii) Tε m×pm is a matrix formed from a p length reference code, (iii) Lε p×n is an optical signal that comprises n column vectors ξ ω,1 , . . . , ξ ω,n , (iv) environmental vectors ξ ω,1 , . . . , ξ ω,n correspond to light reflecting from n different light sources, (v) xε m×1 , (vi) m is a number of samples taken, (vii) positive semidefinite matrix Σ is a covariance matrix of noise, and (viii) η is an additive noise vector with zero mean and variance σ 2 .

9. The method of claim 7 , wherein:
(a) before the data is stored in the electronic memory device, a computer performs an algorithm to calculate the modulation signals and the reference signal; and
(b) the algorithm includes performing semidefinite optimization to solve

M
*

=

arg
⁢

⁢
min

Q

⁢

⁢
t
⁢

⁢

r
⁡

(
Q
)

⁢

⁢

s
.
t

1
≽

vec
⁡

(
M
)

≽
0

,

Q
≽

(

M
T

⁢

∑

-
1

⁢
M

)

-
1

where (i) Q is an auxilary variable; (ii) Q−(M T Σ −1 M) −1 is a positive semidefinite matrix; (iii) noise matrix Σ=diag(σ 1 2 , . . . , σ m 2 ), (iv) σ 1 2 , . . . , σ m 2 are noise sources, (v) M=TL, (vi) Tε m×pm is a matrix formed from a p length reference code, (vii) Lε p×n is an optical signal that comprises n column vectors ξ ω,1 , . . . , ξ ω,n , (viii) environmental vectors ξ ω,1 , . . . , ξ ω,n correspond to light reflecting from n different light sources, (ix) m is a number of samples taken, (x) tr is a trace operator, and (xi) denotes a component-wise inequality for vectors and linear matrix inequalities for matrices.

10. The method of claim 7 , wherein:
(a) before the data is stored in the electronic memory device, a computer performs an algorithm to calculate the modulation signals and the reference signal; and
(b) the algorithm includes performing non-negative matrix factorization to solve

Assignees

• Kadambi Achuta
• Bhandari Ayush
• Raskar Ramesh
• Massachusetts Inst Technology

Inventors

• Kadambi Achuta
• Bhandari Ayush
• Raskar Ramesh

Classifications

• H04N23/56 Primary

  provided with illuminating means · CPC title

• G01S7/4815

  using multiple transmitters · CPC title

• G06T2207/10028

  Range image; Depth image; 3D point clouds · CPC title

• G01S17/894

  Three-dimensional [3D] imaging with simultaneous measurement of time-of-flight at a two-dimensional [2D] array of receiver pixels, e.g. time-of-flight cameras or flash lidar · CPC title

• H04N5/2226 Primary

  Determination of depth image, e.g. for foreground/background separation (determining depth by image analysis in general G06T7/50; segmentation by image analysis in general G06T7/10) · CPC title