Method and apparatus for automatic calibration of RGBZ sensors utilizing epipolar geometry and scanning beam projector

Patent metadata

Field	Value
Publication number	US-10531073-B2
Application number	US-201615157389-A
Country	US
Kind code	B2
Filing date	May 17, 2016
Priority date	Mar 17, 2016
Publication date	Jan 7, 2020
Grant date	Jan 7, 2020

Abstract

Using one or more patterned markers inside the projector module of a three-dimensional (3D) camera to facilitate automatic calibration of the camera's depth sensing operation. The 3D camera utilizes epipolar geometry-based imaging in conjunction with laser beam point-scans in a triangulation-based approach to depth measurements. A light-sensing element and one or more reflective markers inside the projector module facilitate periodic self-calibration of camera's depth sensing operation. To calibrate the camera, the markers are point-scanned using the laser beam and the reflected light is sensed using the light-sensing element. Based on the output of the light-sensing element, the laser's turn-on delay is adjusted to perfectly align a laser light spot with the corresponding reflective marker. Using reflective markers, the exact direction and speed of the scanning beam over time can be determined as well. The marker-based automatic calibration can periodically run in the background without interfering with the normal camera operation.

First claim

What is claimed is:

1. An imaging unit, comprising:
a projector module having an enclosure that includes:
a light source operable to perform a one-dimensional (1D) point scan of a three-dimensional (3D) object along a plurality of scanning lines, wherein the point scan projects a sequence of light spots on a surface of the 3D object through an opening in the enclosure, wherein the opening defines boundaries of a field of scan on the surface of the 3D object in which the light spots are projected along the plurality of scanning lines, and wherein the opening is such that each of the plurality of scanning lines extends inside the enclosure beyond the field of scan;
at least one first reflective marker on a first side of the opening inside the enclosure, wherein each first reflective marker is present along at least one scanning line and outside of the field of scan;
a photo sensor operable to sense light reflected from each first reflective marker that is illuminated by the light source when the light source is activated to project light spots beyond the field of scan, the photo sensor operable to generate an output signal upon sensing the light reflected from a corresponding first reflective marker that is illuminated by the light source; and
an image sensor unit operatively coupled to the projector module to send an enable signal to the projector module to activate the light source to project one or more light spots on the corresponding first reflective marker, receive the output signal from the photo sensor, analyze the output signal received from the photo sensor, determine an Area-Under-the-Curve (AUC) of the output signal, and adjust a timing delay of the enable signal to maximize the AUC of the output signal to control future activation of the light source to improve physical placement of a subsequent light spot on the corresponding first reflective marker when the subsequent light spot is projected by the light source upon future activation.

2. The imaging unit of claim 1 , wherein the light source is operable to perform the point scan in a raster scan pattern.

3. The imaging unit of claim 1 , wherein each first reflective marker is aligned with the opening.

4. The imaging unit of claim 1 , wherein at least one first reflective marker includes two or more first reflective markers on the first side of the opening inside the enclosure, and wherein each of the two or more first reflective markers is present along at least one scanning line outside of the field of scan.

5. The imaging unit of claim 1 , wherein the enclosure further includes:
at least one second reflective marker either above or below a corresponding first reflective marker on the first side of the opening inside the enclosure, wherein each second reflective marker is present along at least one scanning line and outside of the field of scan,
and wherein the photo sensor is operable to sense light reflected from each second reflective marker that is illuminated by the light source when the light source is activated to project light spots beyond the field of scan.

6. The imaging unit of claim 1 , wherein the enclosure further includes at least one of the following:
at least one second reflective marker on a second side of the opening inside the enclosure, wherein the second side is opposite of the first side, and wherein each second reflective marker is present along at least one scanning line and outside of the field of scan; and
at least one third reflective marker either above or below a corresponding second reflective marker on the second side of the opening inside the enclosure, wherein each third reflective marker is present along at least one scanning line and outside of the field of scan,
and wherein the photo sensor is operable to sense light reflected from each second and third reflective markers that are illuminated by the light source when the light source is activated to project light spots beyond the field of scan.

7. The imaging unit of claim 1 , wherein the image sensor unit is operative to further perform the following:
repeatedly send a sequence of enable signals to the projector module and receive corresponding output signals from the photo sensor;
continue to analyze each received output signal and to thereby adjust a timing delay of each newly-transmitted enable signal in the sequence of enable signals; and
conclude repeated transmission of multiple enable signals when a most-recent light spot projected by the light source is aligned with the corresponding first reflective marker.

8. The imaging unit of claim 1 , wherein the image sensor unit includes:
a plurality of pixels arranged in a two-dimensional (2D) pixel array forming an image lane, wherein a row of pixels in the 2D pixel array forms an epipolar line of one of the plurality of scanning lines, and wherein each pixel in the row of pixels is operative to detect a corresponding light spot in a portion of the sequence of light spots projected by the light source along the one of the plurality of scanning lines.

9. The imaging unit of claim 1 , wherein the light source is one of the following:
a laser operating in conjunction with a resonating-type Micro Electro-Mechanical Systems (MEMS) scanner;
a laser operating in conjunction with an electro-static MEMS scanner; and
an X-Y addressable light source.

10. A method, comprising:
performing a one-dimensional (1D) point scan of a three-dimensional (3D) object along a first scanning line, wherein the point scan projects a sequence of light spots on a surface of the 3D object through an opening in an enclosure, wherein the opening defines boundaries of a field of scan on the surface of the 3D object in which the light spots are projected along the first scanning line, and wherein the opening is such that the first scanning line extends inside the enclosure beyond the field of scan;
projecting one or more light spots on a first reflective marker that is present adjacent to the opening inside the enclosure along the first scanning line and outside of the field of scan;
sensing light reflected from the first reflective marker;
generating an output signal based on the sensed light;
based on the sensed light, determine an Area-Under-the-Curve (AUC) of the output signal; and
adjusting placement of the one or more light spots to be projected on the first reflective marker in a future point scan to maximize the AUC of the output signal.

11. The method of claim 10 , wherein projecting the one or more light spots includes:
triggering a light source inside the enclosure to project the one or more light spots on the first reflective marker,
wherein sensing the light includes:
using a photo sensor inside the enclosure to sense the light reflected from the first reflective marker and to generate an output signal upon sensing the reflected light, and

wherein adjusting the placement includes:
analyzing the output signal; and
based on the analysis of the output signal, modifying a timing of the triggering of the light source during the future point scan so as to adjust the placement of the one or more light spots to be projected on the first reflective marker.

12. The method of claim 11 , wherein analyzing the output signal includes:
comparing rise and fall times of the output signal to determine a symmetry of the output signal around a midpoint, and
wherein the method further includes:
ceasing modification of the timing of the triggering of the light source upon determining the symmetry of the output signal around the midpoint.

13. The method of claim 10 , further comprising:
performing the point scan along a second scanning line within the fi

Assignees

• Samsung Electronics Co Ltd

Inventors

• Ovsiannikov Ilia

Classifications

• G01C25/00

  Manufacturing, calibrating, cleaning, or repairing instruments or devices referred to in the other groups of this subclass (testing, calibrating or compensating compasses G01C17/38) · CPC title

• G01C3/08

  Use of electric radiation detectors · CPC title

• H04N13/254

  in combination with electromagnetic radiation sources for illuminating objects · CPC title

• H04N13/246 Primary

  Calibration of cameras · CPC title

• H04N13/271

  wherein the generated image signals comprise depth maps or disparity maps · CPC title