Time-of-flight simulation data training circuitry, time-of-flight simulation data training method, time-of-flight simulation data output method, time-of-flight simulation data output circuitry

Patent metadata

Field	Value
Publication number	US-12061265-B2
Application number	US-202117527191-A
Country	US
Kind code	B2
Filing date	Nov 16, 2021
Priority date	Nov 23, 2020
Publication date	Aug 13, 2024
Grant date	Aug 13, 2024

Abstract

The present disclosure generally pertains to time-of-flight simulation data training circuitry, configured to:
obtain time-of-flight camera model data based on a time-of-flight camera model modelling a real time-of-flight camera of a real time-of-flight camera type; obtain real time-of-flight data from at least one real time-of-flight camera of the real time-of-flight camera type; determine a difference between the real time-of-flight data and the time-of-flight camera model data; and update, based on the determined difference, the time-of-flight camera model for generating simulated time-of-flight data representing the real time-of-flight camera.

First claim

The invention claimed is:

1. Time-of-flight simulation data training circuitry, configured to:
obtain time-of-flight camera model data based on a time-of-flight camera model modelling a real time-of-flight camera of a real time-of-flight camera type, wherein the time-of-flight camera model data originate from a ground truth depth of simulated scenes which are given as ground truth to a pretrained denoiser;
obtain real time-of-flight data from at least one real time-of-flight camera of the real time-of-flight camera type;
determine a difference between the real time-of-flight data and the time-of-flight camera model data; and
update, based on the determined difference, the time-of-flight camera model for generating simulated time-of-flight data representing the real time-of-flight camera.

2. The time-of-flight simulation data training circuitry of claim 1 , wherein the time-of-flight camera model data is obtained from a time-of-flight camera model neural network.

3. The time-of-flight simulation data training circuitry of claim 2 , wherein the time-of-flight camera model neural network is a convolutional neural network.

4. The time-of-flight simulation data training circuitry of claim 1 , wherein the difference between the real time-of-flight data and the time-of-flight camera model data is determined in a discriminative neural network.

5. The time-of-flight simulation data training circuitry of claim 1 , further configured to:
obtain modelled time-of-flight camera data.

6. The time-of-flight simulation data training circuitry of claim 5 , further configured to:
transform the modelled time-of-flight camera data into the time-of-flight camera model data.

7. The time-of-flight simulation data training circuitry of claim 6 , wherein the transforming is carried out with a time-of-flight camera model neural network.

8. The time-of-flight simulation data training circuitry of claim 5 , wherein the modelled time-of-flight camera data is obtained from time-of-flight model-based simulation circuitry.

9. The time-of-flight simulation data training circuitry of claim 1 , wherein the real time-of-flight data is obtained from a plurality of time-of-flight cameras of the real time-of-flight camera type for determining a statistical difference of the real time-of-flight camera type and the time-of-flight camera model.

10. A time-of-flight simulation data training method, comprising:
obtaining time-of-flight camera model data based on a time-of-flight camera model modelling a real time-of-flight camera of a real time-of-flight camera type, wherein the time-of-flight camera model data originate from a ground truth depth of simulated scenes which are given as ground truth to a pretrained denoiser;
obtaining real time-of-flight data of at least one real time-of-flight camera of the real time-of-flight camera type;
determining a difference between the real time-of-flight data and the time-of-flight camera model data; and
updating, based on the determined difference, the time-of-flight camera model for generating simulated time-of-flight data representing the real time-of-flight camera.

11. The time-of-flight simulation data training method of claim 10 , wherein the time-of-flight camera model data is obtained from a time-of-flight camera model neural network.

12. The time-of-flight simulation data training method of claim 11 , wherein the time-of-flight camera model neural network is a convolutional neural network.

13. The time-of-flight simulation data training method of claim 10 , wherein the difference between the real time-of-flight data and the time-of-flight camera model data is determined in a discriminative neural network.

14. The time-of-flight simulation data training method of claim 13 , wherein the modelled time-of-flight camera data is obtained from time-of-flight model-based simulation circuitry.

15. The time-of-flight simulation data training method of claim 10 , further comprising:
obtaining modelled time-of-flight camera data.

16. The time-of-flight simulation data training method of claim 15 , further comprising:
transforming the modelled time-of-flight camera data into the time-of-flight camera model data.

17. The time-of-flight simulation data training method of claim 16 , wherein the transforming is carried out with a time-of-flight camera model neural network.

18. The time-of-flight simulation data training method of claim 10 , wherein the real time-of-flight data is obtained from a plurality of time-of-flight cameras of the real time-of-flight camera type for determining a statistical difference of the real time-of-flight camera type and the time-of-flight camera model.

19. A time-of-flight simulation data output method, comprising:
inputting modelled time-of-flight camera data into a time-of-flight simulation neural network configured to output simulated time-of-flight data representing a real time-of-flight camera of a real time-of-flight camera type, wherein the time-of-flight simulation neural network is trained to generate the simulated time-of-flight data based on a trained difference between time-of-flight camera model data being obtained based on a time-of-flight camera model, modelling the real time-of-flight camera, and real time-of-flight data obtained from at least one real time-of-flight camera of the real time-of-flight camera type,
wherein the time-of-flight camera model data originate from a ground truth depth of simulated scenes which are given as ground truth to a pretrained denoiser.

20. The time-of-flight simulation data output method of claim 19 , wherein the time-of-flight simulation neural network is a convolutional neural network.

21. The time-of-flight simulation data output method of claim 19 , further comprising:
obtaining the modelled time-of-flight camera data from time-of-flight model-based simulation circuitry.

22. The time-of-flight simulation data output method of claim 21 , further comprising:
transforming the modelled time-of-flight camera data into the simulated time-of-flight data; and
outputting the simulated time-of-flight data.

23. Time-of-flight simulation data circuitry configured to:
provide a time-of-flight simulation neural network, configured to output simulated time-of-flight data representing a real time-of-flight camera of a real time-of-flight camera type, wherein the time-of-flight simulation neural network is trained to generate the simulated time-of-flight data based on a trained difference between time-of-flight camera model data being obtained based on a time-of-flight camera model, modelling the real time-of-flight camera, and real time-of-flight data obtained from at least one real time-of-flight camera of the real time-of-flight camera type,
wherein the time-of-flight camera model data originate from a ground truth depth of simulated scenes which are given as ground truth to a pretrained denoiser.

24. The time-of-flight simulation data circuitry of claim 23 , wherein the time-of-flight camera model is based on a time-of-flight camera model neural network.

25. The time-of-flight simulation data circuitry of claim 24 , wherein the time-of-flight camera model neural network is a convolutional neural network.

26. The time-of-flight simulation data circuitry of claim 23 , further configured to:
obtain the modelled time-of-flight camera data from time-of-flight model-based simulation circuitry.

27. T

Assignees

• Sony Semiconductor Solutions Corp

Inventors

• Agresti Gianluca
• Schäfer Henrik
• Incesu Yalcin
• Sartor Piergiorgio
• Zanuttigh Pietro

Classifications

• G06N3/094

  Adversarial learning · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06T2207/20081

  Training; Learning · CPC title

• G06T2207/10028

  Range image; Depth image; 3D point clouds · CPC title