Systems, methods, and media for single photon depth imaging with improved efficiency using learned compressive representations

Patent metadata

Field	Value
Publication number	US-2025035750-A1
Application number	US-202418787781-A
Country	US
Kind code	A1
Filing date	Jul 29, 2024
Priority date	Jul 27, 2023
Publication date	Jan 30, 2025
Grant date	—

Abstract

In accordance with some embodiments, various examples of systems, methods, and media for single photon depth imaging with improved efficiency using learned compressive representations are provided. For example, systems and methods herein may detect a photon arrival based on a signal from a detector at a given position and time bin. A compressed histogram comprising K stored values, representing bins of the compressed histogram based on K values in a code word calculated based on the time bin and position and K coding tensors. Such systems and methods may be utilized for, among other uses, determining depth in a scene being imaged by the detector.

First claim

What is claimed is:

1 . A system for determining a depth in a scene, comprising:
a light source; an array comprising a plurality of detectors configured to detect arrival of individual photons; at least one processor that is programmed to:
(a) detect, based on a signal from a detector of the plurality of detectors, a photon arrival,
wherein the detector of the plurality of detectors has a position p′;

(b) determine a time bin i associated with the photon arrival, wherein the time bin is in a range from 1 to N t where N t is a total number of time bins;
(c) update a compressed histogram comprising K stored values representing bins of the compressed histogram based on K values in a code word calculated based on the time bin i and the position p′ and K coding tensors,
wherein each coding tensor of the K coding tensors is different than each other coding tensor; and

(d) perform an imaging task based on the K values of the compressed histogram.

2 . The system of claim 1 , wherein each of the plurality of detectors comprises a single photon avalanche diode (SPAD).

3 . The system of claim 1 , wherein each of the coding tensors has a size M t ×M r ×M c , and wherein the at least one processor is further programmed to:
estimate depth values for M r ×M c detectors using the K values of the compressed histogram.

4 . The system of claim 1 , wherein (a) to (c) are performed by circuitry that is implemented on a same chip as the plurality of detectors.

5 . The system of claim 4 , wherein (d) is performed by circuitry that is implemented on a different chip than the plurality of detectors.

6 . The system of claim 1 , wherein the at least one processor is further programmed to:
perform a dot product operation between a one-hot matrix and the K coding tensors,
wherein the one-hot matrix has a 1 at a position corresponding to position i,p′ within a block b of positions having a size M t ×M r ×M c , where i is the time bin i and p′ is a position;
transfer the K values of the compressed histogram from a chip on which the plurality of detectors are implemented to a second chip; and perform an unfiltered backprojection of the K values of the compressed histogram using the K coding tensors, thereby generating an M t ×M r ×M c matrix of values, wherein to perform the imaging task, the at least one processor is programmed to:
estimate a depth value for the detector based on the M r ×M c matrix of values.

7 . The system of claim 1 , wherein the K coding tensors change over time based on the scene.

8 . The system of claim 6 , wherein to perform the convolution between the one-hot matrix and the K coding tensors, the at least one processor is programmed to: perform a lookup based on each dot product of the convolution, wherein the one-hot matrix comprises a single element having a 1.

9 . The system of claim 6 , wherein the at least one processor is further programmed to:
estimate the depth value for the detector based on the M t ×M r ×M c matrix of values using a convolutional neural network (CNN),
wherein a first layer of the CNN comprises the convolution between the one-hot matrix and the K coding tensors, and other layers of the CNN are trained to determine depth values from an N t ×N r ×N c matrix of values,
wherein N t ×N r ×N c comprises a plurality of matrices each having a size of M t ×M r ×M c .

10 . The system of claim 9 , wherein at least some of the weights of the K coding tensors were trained using a CNN training process.

11 . The system of claim 1 , wherein the at least one processor is further programmed to:
perform (a) to (d) for each of the plurality of detectors.

12 . The system of claim 1 , wherein M t is less than or equal to N t .

13 . The system of claim 1 , wherein each of the K coding tensors is expressible as an outer product of two tensors C k temporal , and C k spatial , where C k temporal is a M t ×1×1 tensor, and C k spatial is a 1×M r ×M c tensor.

14 . A method for determining a depth in a scene, comprising:
(a) detecting, based on a signal from a detector of a plurality of detectors, a photon arrival,
wherein the detector of the plurality of detectors has a position p′;
(b) determining a time bin i associated with the photon arrival, wherein the time bin is in a range from 1 to N t where N t is a total number of time bins; (c) updating a compressed histogram comprising K stored values representing bins of the compressed histogram based on K values in a code word calculated based on the time bin i and the position p′ and K coding tensors,
wherein each coding tensor of the K coding tensors is different than each other coding tensor; and
(d) performing an imaging task based on the K values of the compressed histogram.

15 . The method of claim 14 , wherein each of the plurality of detectors comprises a single photon avalanche diode (SPAD).

16 . The method of claim 14 , wherein each of the coding tensors has a size M t ×M r ×M c , the method further comprising:
estimating depth values for M r ×M c detectors using the K values of the compressed histogram.

17 . The method of claim 14 , further comprising:
performing a convolution between a one-hot matrix and the K coding tensors,
wherein the one-hot matrix has a 1 at a position corresponding to position i,p′ within a block b of positions having a size M t ×M r ×M c , where i is the time bin i and p′ is a position;
transferring the K values of the compressed histogram from a chip on which the plurality of detectors are implemented to a second chip; performing, using a processor implemented on the second chip, an unfiltered backprojection of the K values of the compressed histogram using the K coding tensors, thereby generating an M t ×M r ×M c matrix of values; and estimating the depth value for the detector based on the M t ×M r ×M c matrix of values.

18 . The method of claim 14 , wherein each of the K coding tensors is expressible as an outer product of two tensors C k temporal , and C k spatial , where C k temporal is a M t ×1×1 tensor, and C k spatial is a 1×M r ×M c tensor.

19 . A system for generating compressed single-photon histograms, comprising:
a light source; an array comprising a plurality of detectors configured to detect arrival of individual photons; at least one processor that is programmed to:
(a) detect, based on a signal from a detector of the plurality of detectors, a photon arrival,
wherein the detector of the plurality of detectors has a position p′;

(b) determine a time bin i associated with the photon arrival, wherein the time bin is in a range from 1 to N t where N t is a total number of time bins;
(c) update a compressed histogram comprising K stored values representing bins of the compressed histogram based on K values in a code word calculated based on the time bin i and the position p′ and K coding tensors,
wherein each coding tensor of the K coding tensors is different than each other coding tensor; and

(d) output the compressed histogram to another processor.

20 . The system of claim 18 , wherein each of the coding tensors has a size M t ×M r ×M c , and wherein the at least one processor is further programmed to perform at least one of the following:
estimate depth values for M r ×M c detectors using the K values of the compressed histogram;
perform a 3D object detection for an object of a scene represented in the signal from the plurality of detectors;
perform a 3D image segmentation

Assignees

• Wisconsin Alumni Res Found

Inventors

• Gutierrez Barragan Felipe
• Velten Andreas
• Mu Fangzhou
• Li Yin
• Gupta Mohit
• Ingle Atul

Classifications

• G01S17/10

  using transmission of interrupted, pulse-modulated waves (determination of distance by phase measurements G01S17/32) · CPC title

• G01S7/4865

  Time delay measurement, e.g. time-of-flight measurement, time of arrival measurement or determining the exact position of a peak (peak detection in noise, signal conditioning G01S7/487) · CPC title

• G01S7/4863

  Detector arrays, e.g. charge-transfer gates · CPC title

• G01S7/4808 Primary

  Evaluating distance, position or velocity data · CPC title

• G01S17/42

  Simultaneous measurement of distance and other co-ordinates (indirect measurement G01S17/46) · CPC title