Apparatuses and methods for training a machine learning network for use with a time-of-flight camera

Patent metadata

Field	Value
Publication number	US-2021166124-A1
Application number	US-202017110330-A
Country	US
Kind code	A1
Filing date	Dec 3, 2020
Priority date	Dec 3, 2019
Publication date	Jun 3, 2021
Grant date	—

Abstract

The present disclosure proposes a concept of training a machine learning network for use with a ToF camera. Based on a predefined synthetic scene, it is simulated a ground truth time resolved illumination return signal for a light pulse emitted from the ToF camera to the synthetic scene and scattered back from the synthetic scene to the ToF camera. The synthetic scene comprises a plurality of scene points with known distances between each of the scene points and the ToF camera. Based on the ground truth time resolved illumination return signal and a simulation model of the ToF camera, it is then simulated an output signal of at least one ToF pixel capturing the synthetic scene. Based on the simulated output signal of the ToF pixel and the ground truth time resolved illumination return signal, weights of the machine learning network are adjusted to cause the machine learning network to map the simulated output signal of the ToF pixel to an output time resolved illumination return signal approximating the ground truth time resolved illumination return signal.

First claim

1 . Method of training a machine learning network for use with a ToF camera, the method comprising:
simulating, based on at least one predefined synthetic scene, a ground truth time resolved illumination return signal for a light pulse emitted from the ToF camera to the synthetic scene and scattered back from the synthetic scene to the ToF camera, wherein the synthetic scene comprises a plurality of scene points with known distances between each of the scene points and the ToF camera; simulating, based on the ground truth time resolved illumination return signal and a simulation model of the ToF camera, an output signal of at least one ToF pixel capturing light from the synthetic scene; and adjusting, based on the simulated output signal of the ToF pixel and the ground truth time resolved illumination return signal, weights of the machine learning network to cause the machine learning network to map the simulated output signal of the ToF pixel to an output time resolved illumination return signal approximating the ground truth time resolved illumination return signal.

2 . The method of claim 1 , wherein simulating the output signal comprises simulating the output signal of the ToF pixel for a plurality of different light modulation frequencies to obtain a respective output signal for each of the different light modulation frequencies, and wherein the weights of the machine learning network are adjusted based on the simulated output signals for the different light modulation frequencies as input to the machine learning network and based on the ground truth multipath propagation delay profile.

3 . The method of claim 1 , wherein simulating the ground truth time resolved illumination return signal comprises, based on transient rendering, simulating non-line-of-sight light components scattered back from the synthetic scene to the ToF camera.

4 . The method of claim 1 , wherein adjusting the weights of the machine learning network comprises minimizing a difference between the output time resolved illumination return signal and the ground truth time resolved illumination return signal.

5 . The method of claim 4 , wherein the difference is measured according to the earth mover's distance, EMD.

6 . The method of claim 1 , wherein the method of training is repeated for each of the plurality of scene points and/or a plurality of different predefined synthetic scenes.

7 . The method of claim 1 , wherein the simulation model of the ToF camera models specific camera properties including at least one of exact illumination shape, sensor sensitivity shape, and noise levels.

8 . The method of claim 1 , wherein the machine learning network comprises a convolutional network architecture.

9 . The method of claim 1 , wherein the ToF camera is an indirect ToF camera.

10 . Method for operating a ToF camera, the method comprising:
capturing a scene with a ToF pixel matrix of the ToF camera; feeding output signals of the ToF pixel matrix into a machine learning network trained according to the method of claim 1 ; and providing, at an output of the machine learning network, time resolved illumination return signals associated with pixels of the ToF pixel matrix.

11 . A computer program having a program code for performing a method of claim 1 , when the computer program is executed on a programmable hardware device.

12 . A ToF camera comprising a machine learning network trained according to the method of claim 1 .

13 . The ToF camera according to claim 12 , wherein an input layer of the machine learning network is coupled to an output of a ToF pixel matrix of the ToF camera.

14 . The ToF camera according to claim 12 , wherein the machine learning network is configured to output a time resolved illumination return signal associated with each pixel of the ToF pixel matrix.

15 . The ToF camera according to claim 12 , wherein the ToF camera is an indirect ToF camera.

Assignees

• Sony Semiconductor Solutions Corp

Inventors

• Schäfer Henrik
• Buratto Enrico
• Agresti Gianluca
• Zanuttigh Pietro

Classifications

• G06N3/0475

  Generative networks · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06N3/094

  Adversarial learning · CPC title

• G01S7/497

  Means for monitoring or calibrating · CPC title