Method for epipolar time of flight imaging

Patent metadata

Field	Value
Publication number	US-11425357-B2
Application number	US-201816468617-A
Country	US
Kind code	B2
Filing date	Jan 19, 2018
Priority date	Feb 13, 2015
Publication date	Aug 23, 2022
Grant date	Aug 23, 2022

Abstract

Energy-efficient epipolar imaging is applied to the ToF domain to significantly expand the versatility of ToF sensors. The described system exhibits 15+ m range outdoors in bright sunlight; robustness to global transport effects such as specular and diffuse inter-reflections; interference-free 3D imaging in the presence of many ToF sensors, even when they are all operating at the same optical wavelength and modulation frequency; and blur- and distortion-free 3D video in the presence of severe camera shake. The described embodiments are broadly applicable in consumer and robotics domains.

First claim

We claim:

1. A system comprising:
a modulated light source for projecting a steerable sheet of modulated light into a field-of-view;
a sensor having a selectable region of interest, the modulated light source and sensor in a rectified stereo configuration such that the steerable sheet of modulated light lies in an epipolar plane between the modulated light source and the sensor; and
a microcontroller for synchronizing the modulated light source and the sensor such that the selected region of interest of the sensor is set to a portion of the field-of-view containing the epipolar plane currently illuminated by the steerable sheet of modulated light.

2. The system of claim 1 , wherein the modulated light source comprises:
a laser source;
an optical element, configured to generate the steerable sheet of modulated from a collimated output of the laser source; and
a means for steering the sheet of modulated light along a series of epipolar planes between the projector and the sensor.

3. The system of claim 2 wherein the means for steering the sheet of modulated light is selected from a group comprising a rotatable galvomirror and a MEMS mirror.

4. The system of claim 2 , wherein the sensor is a continuous wave time-of-flight camera having a controllable region of interest.

5. The system of claim 4 , wherein the steerable sheet of modulated light illuminates a single row of pixels in the sensor, and further wherein the controllable region of interest of the sensor is set to sense the single row of illuminated pixels.

6. The system of claim 1 , wherein the sensor captures at least two images from each illuminated epipolar plane.

7. The system of claim 1 wherein the steerable sheet of modulated light is modulated as a repeating wave and further wherein a sensed depth may be calculated for each sensed pixel based on phases of the returned reflections.

8. The system of claim 7 wherein a depth map of the entire field of view is created based upon the depth calculated for each sensed pixel from each illuminated epipolar plane within the field-of-view.

9. The system of claim 2 , wherein the microcontroller reads data from the sensor regarding the previously illuminated epipolar plane while the means for steering the sheet of modulated light moves the steerable sheet of light such as to illuminate a next epipolar plane in the series of epipolar planes.

10. A method comprising:
projecting a sheet of modulated light along a series of epipolar planes defined by a modulated light source and a sensor placed in a rectified stereo configuration, the series of epipolar planes defining a field-of-view; and
imaging a single row of illuminated pixels in an epipolar plane currently illuminated by the sheet of modulated light; and
synchronizing the modulated light source and the sensor, such that a region of interest of the sensor is set to a portion of the field-of-view containing an epipolar plane currently illuminated by the sheet of modulated light.

11. The method of claim 10 , further comprising:
determining a depth of each pixel in the single row of illuminated pixels;
wherein determining the depth of each pixel comprises calculating the depth of each pixel based on a phase of the reflected light from each pixel.

12. The method of claim 11 wherein calculating the depth of each pixel in an illuminated epipolar plane further comprises determining a difference in phase of reflected light contained in two or more separate images of the illuminated epipolar plane.

13. The method of claim 11 , further comprising creating a depth map based on the depth of each pixel in each illuminated epipolar plane within the defined field-of-view.

14. The method of claim 10 wherein the epipolar planes in the field-of-view are illuminated in a varying order.

15. The method of claim 10 wherein the modulated light source comprises:
a laser source;
an optical element, configured to generate a light sheet from a collimated output of the laser source; and
a means for steering the light sheet of laser light along a series of epipolar planes between the modulated light source and the sensor.

16. A non-transitory computer-readable media containing software that when executed, performs the functions of:
projecting a sheet of modulated light along a series of epipolar planes defined by a modulated light source and a sensor placed in a rectified stereo configuration, the series of epipolar planes defining a field-of-view;
imaging a single row of illuminated pixels from each epipolar plane in the series of epipolar planes as each epipolar plane is illuminated by the projected sheet of modulated light; and
synchronizing a modulated light source projecting the sheet of modulated light and a sensor imaging the single row of illuminated pixels, such that a region of interest of the sensor corresponds to a currently illuminated epipolar plane.

17. The non-transitory computer-readable media of claim 16 , wherein the software performs the further function of:
determining a depth of each pixel in the single row of illuminated pixels;
wherein determining the depth of each pixel comprises calculating the depth of each pixel in an illuminated epipolar plane by determining a difference in phase of reflected light contained in two or more separate images of the illuminated epipolar plane.

18. The non-transitory computer-readable media of claim 16 wherein the epipolar planes in the field-of-view are illuminated in a varying order.

19. The method of claim 16 wherein the modulated light source comprises:
a laser source;
an optical element, configured to generate the sheet of modulated light from a collimated output of the laser source; and
a means for steering the sheet of modulated light along a series of epipolar planes between the modulated light source and the sensor.

20. The non-transitory computer-readable media of claim 17 wherein the software performs the further function of:
creating a depth map based on the depth of each pixel in each illuminated epipolar plane within the defined field-of-view.

Assignees

• Univ Carnegie Mellon
• Governing Council Univ Toronto

Inventors

• Narasimhan Srinivasa
• Achar Supreeth
• Kutulakos Kiriakos
• Bartels Joseph
• Whittaker William

Classifications

• G01S7/4814

  of transmitters alone · CPC title

• H04N13/211

  using temporal multiplexing · CPC title

• G01S17/894 Primary

  Three-dimensional [3D] imaging with simultaneous measurement of time-of-flight at a two-dimensional [2D] array of receiver pixels, e.g. time-of-flight cameras or flash lidar · CPC title

• G01S7/4914

  of detector arrays, e.g. charge-transfer gates · CPC title

• H04N13/271

  wherein the generated image signals comprise depth maps or disparity maps · CPC title