Depth from time-of-flight using machine learning

Patent metadata

Field	Value
Publication number	US-9760837-B1
Application number	US-201615068632-A
Country	US
Kind code	B1
Filing date	Mar 13, 2016
Priority date	Mar 13, 2016
Publication date	Sep 12, 2017
Grant date	Sep 12, 2017

Abstract

A depth detection apparatus is described which has a memory storing raw time-of-flight sensor data received from a time-of-flight sensor. The depth detection apparatus also has a trained machine learning component having been trained using training data pairs. A training data pair comprises at least one simulated raw time-of-flight sensor data value and a corresponding simulated ground truth depth value. The trained machine learning component is configured to compute in a single stage, for an item of the stored raw time-of-flight sensor data, a depth value of a surface depicted by the item, by pushing the item through the trained machine learning component.

First claim

The invention claimed is:

1. A depth detection apparatus comprising:
a memory storing raw time-of-flight sensor data received from a time-of-flight sensor; and
a processor comprising a trained machine learning component having been trained using training data pairs, a training data pair comprising at least one simulated raw time-of-flight sensor frame and a corresponding simulated ground truth depth map;
the trained machine learning component configured to compute in a single stage, for an item of the stored raw time-of-flight sensor data, a depth map of a surface depicted by the item, by pushing the item through the trained machine learning component.

2. The apparatus of claim 1 the trained machine learning component having been trained using simulated raw time-of-flight sensor frames which incorporate simulated multi-path interference.

3. The apparatus of claim 1 the trained machine learning component having been trained using simulated raw time-of-flight sensor frames computed using a computer graphics renderer which simulates multi-path interference.

4. The apparatus of claim 1 the trained machine learning component having been trained using simulated raw time-of-flight sensor frames comprising, for an individual pixel, weighted intensity values at different depths potentially depicted by the pixel.

5. The apparatus of claim 1 the trained machine learning component having been trained using simulated raw time-of-flight sensor frames where information about an exposure profile of the time-of-flight sensor is combined with the simulated raw time-of-flight sensor frames.

6. The apparatus of claim 1 the trained machine learning component having been trained using simulated raw time-of-flight sensor frames where information about sensor noise of the time-of-flight sensor is combined with the simulated raw time-of-flight sensor frames.

7. The apparatus of claim 1 the trained machine learning component having been trained using simulated raw time-of-flight sensor frames computed using a computer graphics renderer from a plurality of instances of a parametric 3D environment model, where the instances of the parametric 3D environment model are computer generated automatically at random.

8. The apparatus of claim 7 where parameters of the parametric 3D environment model comprise one or more of: geometry of an object in the 3D environment model, position of an object in the 3D environment model, presence of an object in the 3D environment model, orientation of an object in the 3D environment model, surface materials and reflectivity, ambient illumination.

9. The apparatus of claim 1 wherein a training data pair comprises a frame of simulated raw time-of-flight sensor data values and a corresponding simulated ground truth depth map.

10. The apparatus of claim 1 , the trained machine learning component having been trained using simulated raw time-of-flight sensor frames computed using a computer graphics renderer for a plurality of randomly selected viewpoints of the time-of-flight sensor, and where any of the viewpoints which are within a threshold distance of a surface in a 3D environment model used by the computer graphics renderer are omitted.

11. The apparatus of claim 1 the trained machine learning component having been trained using simulated raw time-of-flight sensor frames aggregated over a neighborhood of a pixel, where the neighborhood is a spatial neighborhood, or a temporal neighborhood, or a spatial and temporal neighborhood.

12. The apparatus of claim 1 where the trained machine learning component is a pixel independent regressor.

13. The apparatus of claim 1 where the trained machine learning component is regressor which takes into account relationships between pixels of the stored time-of-flight sensor data.

14. The apparatus of claim 1 where the trained machine learning component is a convolutional neural network and where each training data pair comprises a frame of simulated raw time-of-flight sensor data and a ground truth depth map.

15. The apparatus of claim 1 where the trained machine learning component is at least partially implemented using hardware logic selected from any one or more of: a field-programmable gate array, an application-specific integrated circuit, an application-specific standard product, a system-on-a-chip, a complex programmable logic device, a graphics processing unit.

16. A depth detection apparatus comprising:
a memory storing frames of raw time-of-flight sensor data received from a time-of-flight sensor; and
a trained machine learning component having been trained using training data pairs, a training data pair comprising a simulated raw time-of-flight sensor frame and a corresponding simulated ground truth depth map;
the trained machine learning component configured to compute in a single stage, for a frame of the stored raw time-of-flight sensor data, a depth map of surfaces depicted by the frame, by pushing the frame through the trained machine learning component.

17. The apparatus of claim 16 where the trained machine learning component is configured to operate in real time by computing the depth maps at a rate which is equivalent to or faster than a frame rate of the time-of-flight sensor.

18. The apparatus of claim 16 where the trained machine learning component comprises a convolutional neural network.

19. The apparatus of claim 16 where the trained machine learning component comprises a pixel independent regressor which is a regressor that does not take into account relationships between pixels of a time-of-flight sensor frame.

20. A computer-implemented method comprising:
storing, at a memory, raw time-of-flight sensor data received from a time-of-flight sensor; and
operating, by a processor, a trained machine learning component having been trained using training data pairs, a training data pair comprising at least one simulated raw time-of-flight sensor frame and a corresponding simulated ground truth depth value;
wherein operating the trained machine learning component comprises computing, in a single stage, for an item of the stored raw time-of-flight sensor data, a depth map of a surface depicted by the item, by pushing the item through the trained machine learning component.

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Nowozin Sebastian
• Adam Amit
• Mazor Shai
• Yair Omer

Classifications

• G06V10/774

  Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting · CPC title

• G06F18/214

  Generating training patterns; Bootstrap methods, e.g. bagging or boosting · CPC title

• G01S17/36

  with phase comparison between the received signal and the contemporaneously transmitted signal · CPC title

• G01S7/4808

  Evaluating distance, position or velocity data · CPC title

• G01S17/66

  Tracking systems using electromagnetic waves other than radio waves · CPC title