Temporal time-of-flight

Patent metadata

Field	Value
Publication number	US-10229502-B2
Application number	US-201615015065-A
Country	US
Kind code	B2
Filing date	Feb 3, 2016
Priority date	Feb 3, 2016
Publication date	Mar 12, 2019
Grant date	Mar 12, 2019

Abstract

A depth detection apparatus is described which has a memory and a computation logic. The memory stores frames of raw time-of-flight sensor data received from a time-of-flight sensor, the frames having been captured by a time-of-flight camera in the presence of motion such that different ones of the frames were captured using different locations of the camera and/or with different locations of an object in a scene depicted in the frames. The computation logic has functionality to compute a plurality of depth maps from the stream of frames, whereby each frame of raw time-of-flight sensor data contributes to more than one depth map.

First claim

The invention claimed is:

1. A depth detection apparatus comprising:
a memory storing frames of raw time-of-flight sensor data received from a time-of-flight camera, the frames having been captured by the time-of-flight camera in the presence of motion such that different ones of the frames were captured using different locations of the camera or with different locations of an object in a scene depicted in the frames or with both different locations of the camera and different locations of an object in a scene depicted in the frames; and
a hardware processor configured to perform computation logic to compute a plurality of depth maps from the frames of raw time-of-flight sensor data, whereby each frame of raw time-of-flight sensor data contributes to the computation of more than one depth map by using a block of the frames of raw time-of-flight sensor data to compute the more than one depth map, the hardware processor further configured to provide the plurality of depth maps to a system that uses the plurality of depth maps to determine depth information, wherein the computation logic has functionality to carry out inference with respect to a model of temporal time-of-flight, the model of temporal time-of-flight comprising a weighted combination of a static model and a temporal model, the temporal model describing how imaging conditions at individual pixels evolve over time and the static model omitting data about evolution over time.

2. The apparatus of claim 1 wherein the model of temporal time-of-flight describes how raw time-of-flight data is generated by the camera under imaging conditions comprising albedo, illumination and surface depth from the camera.

3. The apparatus of claim 2 where the model is a probabilistic model.

4. The apparatus of claim 3 wherein the model of temporal time-of-flight comprises a prior term expressing knowledge about a trajectory of the time-of-flight camera and about scene geometry.

5. The apparatus of claim 4 wherein the prior term has been empirically determined.

6. The apparatus of claim 2 wherein the functionality to carry out inference comprises a look-up table or regressor trained to be a practical working equivalent of the model of temporal time-of-flight.

7. The apparatus of claim 6 where the look up-table or regressor has been trained using training data generated by the model.

8. The apparatus of claim 1 where the memory stores the frames such that different ones of the frames have been captured using different measurement patterns of the time-of-flight camera, and where a sequence of measurement patterns used by the time-of-flight camera has been specified taking into account the model.

9. The apparatus of claim 1 wherein the computation logic is configured to operate in real time whereby a frame rate of the frames captured by the time-of-flight camera is matched or bettered by a rate at which the depth maps are output by the computation logic.

10. The apparatus of claim 1 wherein the computation logic is configured to output the depth maps at a rate of 30 per second or more.

11. An apparatus comprising:
a memory storing frames of raw time-of-flight sensor data received from a time-of-flight camera, the frames having been captured by the time-of-flight camera in the presence of motion such that different ones of the frames were captured using different locations of the camera and/or with different locations of an object in a scene depicted in the frames; and
a hardware processor configured to perform computation logic to compute a plurality of depth maps from the frames of raw time-of-flight sensor data by carrying out inference with respect to a model of temporal time-of-flight that uses each frame of raw time-of-flight sensor data to compute more than one depth map by using a block of the frames of raw time-of-flight sensor data, wherein the model of temporal time-of-flight comprises a weighted combination of a static model and a temporal model, the temporal model describing how imaging conditions at individual pixels evolve over time and the static model omitting data about evolution over time, where the model of temporal time-of-flight describes how raw time-of-flight data is generated by the camera under imaging conditions comprising albedo, illumination and surface depth from the camera, the processor further configured to provide the plurality of depth maps to a system that uses the plurality of depth maps to determine depth information.

12. The apparatus of claim 11 integral with a time-of-flight camera.

13. The apparatus of claim 11 integral with a mobile computing device.

14. A computer-implemented method comprising:
storing, at a memory, frames of raw time-of-flight sensor data received from a time-of-flight camera, the frames having been captured by the time-of-flight camera in the presence of motion such that different ones of the frames were captured using different locations of the camera or with different locations of an object in a scene depicted in the frames or with both different locations of the camera and different locations of an object in a scene depicted in the frames;
computing a plurality of depth maps from the frames of raw time-of-flight sensor data, whereby each frame of raw time-of-flight sensor data contributes to the computation of more than one depth map by using a block of the frames of raw time-of-flight sensor data to compute the more than one depth map, wherein computing the depth maps comprises carrying out inference with respect to a model of temporal time-of-flight and making a weighted combination of a static model and a temporal model, the temporal model describing how imaging conditions at individual pixels evolve over time and the static model omitting data about evolution over time; and
providing the plurality of depth maps to a system that uses the plurality of depth maps to determine depth information.

15. The method of claim 14 wherein the model describes how raw time-of-flight data is generated by the camera under imaging conditions comprising albedo, illumination and surface depth from the camera.

16. The method of claim 15 where the model is a probabilistic model.

17. The method of claim 14 wherein computing the depth maps comprises using a look-up table or regressor trained to be a practical working equivalent of a model of temporal time-of-flight.

18. The method of claim 17 comprising filling the look up-table or training the regressor using data generated by the model.

19. The method of claim 14 wherein the model of temporal time-of-flight comprises an empirical prior term expressing knowledge about a trajectory of the time-of-flight camera and about scene geometry.

20. The method of claim 14 comprising computing a sequence of measurement patterns to be used by the time-of-flight camera taking into account the model.

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Adam Amit
• Nowozin Sebastian
• Yair Omer
• Mazor Shai
• Schober Michael

Classifications

• G01S17/50

  Systems of measurement based on relative movement of target · CPC title

• G06T2207/10016

  Video; Image sequence · CPC title

• G06T7/521 Primary

  from laser ranging, e.g. using interferometry; from the projection of structured light · CPC title

• G06F3/017

  Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883) · CPC title

• G06T2207/20076

  Probabilistic image processing · CPC title