Fast general multipath correction in time-of-flight imaging

Patent metadata

Field	Value
Publication number	US-9542749-B2
Application number	US-201414148184-A
Country	US
Kind code	B2
Filing date	Jan 6, 2014
Priority date	Jan 6, 2014
Publication date	Jan 10, 2017
Grant date	Jan 10, 2017

Abstract

Fast general multipath correction in time of flight imaging is described, for example, to obtain accurate depth maps at frame rate from a time of flight camera. In various embodiments accurate depth maps are calculated by looking up corrected depth values stored in a look up table. In various embodiments the corrected depth values are highly accurate as they take into account three or more possible light ray paths between the camera and a surface in a scene being imaged. In an example accurate depth maps are computed at a frame rate of a time of flight camera. In an example accurate depth maps are computed in less than 30 milliseconds for an image having over 200,000 pixels using a standard CPU.

First claim

The invention claimed is:

1. A computer-implemented method comprising:
receiving, at a processor, time of flight depth camera amplitude and phase measurements depicting surfaces in a scene;
computing, via the processor, an index of a look up table from the measurements;
using the index to read, via the processor, a corrected depth value from the look up table, the corrected value being a depth of a surface from the camera having been calculated by taking into account at least three possible light ray paths between the camera and a surface in the scene and searching for values of a backscattering vector, wherein the backscattering vector values comprise weights indicating how much contribution is made to the amplitude and phase values observed at a pixel from light associated with a plurality of different distance bins, and the distance bins comprise a distance range from the time of flight camera divided into intervals.

2. A method as claimed in claim 1 comprising repeating the process of claim 1 for a plurality of pixels of an image captured by the time of flight depth camera to obtain a multipath corrected depth map.

3. A method as claimed in claim 2 wherein the depth value that is read from the lookup table is stored in a depth map at a location corresponding to the location of the pixel in the image.

4. A method as claimed in claim 1 wherein the look up table has a number of dimensions which is lower than a number of dimensions of the received time of flight depth camera amplitude and phase measurements.

5. A method as claimed in claim 1 comprising computing the index from the measurements at least by allocating the measurements to one or more of a plurality of bins, where a bin is a specified range of values.

6. A method as claimed in claim 1 comprising computing the index from the measurements by mapping the measurements to a lower dimensional form.

7. A method as claimed in claim 1 , the corrected value having been calculated by searching for values of the backscattering vector which sum to a smallest absolute amount.

8. A method as claimed in claim 1 , the corrected value having been calculated by taking into account noise in the amplitude and phase values.

9. A method as claimed in claim 1 , wherein the search is carried out using a greedy algorithm which allocates a value of the backscattering vector without taking into account future entries of the vector which may influence the allocated value.

10. A method as claimed in claim 1 , wherein the search is carried out using an optimization engine.

11. A method as claimed in claim 1 , the corrected value having been calculated by selecting one of the values of the backscattering vector on the basis of a noise model.

12. A method as claimed in claim 1 , the corrected value having been calculated by selecting one of the values of the backscattering vector on the basis of the position of the values in the backscattering vector.

13. A method as claimed in claim 1 at least partially carried out using hardware logic.

14. A depth map computation apparatus comprising:
a processor arranged to receive time of flight depth camera amplitude and phase measurements depicting a surface in a scene;
a depth map computation logic configured to compute via the processor an index of a look up table from the measurements;
the depth map computation logic configured to read via the processor a corrected depth value from the look up table using the index, the corrected value being a depth of the surface from the camera having been calculated by taking into account at least three possible light ray paths between the camera and the surface in the scene and searching for values of a backscattering vector, wherein the backscattering vector values comprise weights indicating how much contribution is made to the amplitude and phase values observed at a pixel from light associated with a plurality of different distance bins, and the distance bins comprise a distance range from the time of flight camera divided into intervals.

15. An apparatus as claimed in claim 14 comprising a memory storing the look up table.

16. An apparatus as claimed in claim 14 the depth map computation logic being at least partially implemented using hardware logic selected from any one or more of: a field-programmable gate array, a program-specific integrated circuit, a program-specific standard product, a system-on-a-chip, a complex programmable logic device, a graphics processing unit.

17. A method comprising:
receiving, at a processor, time of flight depth camera amplitude and phase measurements depicting surfaces in a scene, the measurements being received at a frame rate of the time of flight camera;
computing, via the processor, indices of a look up table from the measurements;
using the indices to read, via the processor, corrected depth values from the look up table, the corrected values being depths of surfaces from the camera having been calculated so as to reduce the effects of general multipath interference by taking into account at least three possible light ray paths between the camera and a surface in the scene and searching for values of a backscattering vector; and
using the corrected depth values to form, via the processor, a depth map in a time at or below the frame rate of the time of flight camera, wherein the backscattering vector values comprise weights indicating how much contribution is made to the amplitude and phase values observed at a pixel from light associated with a plurality of different distance bins, and the distance bins comprise a distance range from the time of flight camera divided into intervals.

18. A method as claimed in claim 17 wherein general multipath interference comprises interference from two or more light ray paths in addition to a light ray path having no interference.

19. A method as claimed in claim 17 wherein the depth map is formed at a frame rate of a time of flight camera.

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Freedman Daniel
• Krupka Eyal
• Smolin Yoni
• Leichter Ido
• Schmidt Mirko

Classifications

• G01S7/497

  Means for monitoring or calibrating · CPC title

• G06T7/593 Primary

  from stereo images · CPC title

• G01S17/36 Primary

  with phase comparison between the received signal and the contemporaneously transmitted signal · CPC title

• G01S17/894

  Three-dimensional [3D] imaging with simultaneous measurement of time-of-flight at a two-dimensional [2D] array of receiver pixels, e.g. time-of-flight cameras or flash lidar · CPC title

• H04N13/128

  Adjusting depth or disparity · CPC title