Systems, methods, and media for motion adaptive imaging using single-photon image sensor data

Patent metadata

Field	Value
Publication number	US-11539895-B1
Application number	US-202117485978-A
Country	US
Kind code	B1
Filing date	Sep 27, 2021
Priority date	Sep 27, 2021
Publication date	Dec 27, 2022
Grant date	Dec 27, 2022

Abstract

In accordance with some embodiments, systems, methods, and media for motion adaptive imaging using single-photon image sensor data are provided. In some embodiments, the system comprises: an image sensor comprising single-photon detectors in an array; a processor programmed to: receive a sequence of photon frames, each comprising pixels having a value indicative of whether a photon was received during a frame period, each of the pixels corresponds to a pixel location; identify, for each of the pixel locations, changepoints, each indicative of a change in scene brightness; identify a photon frame in the sequence at which at least a threshold change in brightness has occurred based on the changepoints associated with each of the plurality of pixel locations; and generate a series of changepoint frames, wherein each changepoint frame is based on estimated brightness associated with each pixel location at a point in the sequence of photon frames.

First claim

What is claimed is:

1. A system for generating digital images, comprising:
an image sensor comprising a plurality of detectors configured to detect arrival of photons, the plurality of detectors arranged in an array;
at least one processor that is programmed to:
receive, from the image sensor, a series of images, each image in the series of images comprising a plurality of pixels representing brightness of a portion of a scene, each of the plurality of pixels associated with a pixel location (x,y);
identify, for each of the plurality of pixel locations using the series of images, one or more changepoints at which a transition in brightness associated with a pixel between different steady state brightness values occurs;
determine, based on the one or more changepoints associated with the plurality of pixel locations, a first point in the series of images at which at least a threshold portion of the plurality of pixel locations have included at least one changepoint;
generate a first image based on brightness associated with each pixel location at the first point in the series of images;
determine, based on changepoints associated with the plurality of pixel locations, a second point in the series of images at which at least the threshold portion of the plurality of pixel locations have included at least one changepoint that occurred after the first point in the series of images; and
generate a second image based on brightness associated with each pixel location at the second point in the series of images.

2. The system of claim 1 , wherein each of the plurality of detectors is a single-photon avalanche diode (SPAD).

3. The system of claim 2 , wherein each SPAD is configured to record a timestamp at which a single photon was detected.

4. The system of claim 1 , wherein the at least one processor is further programmed to:
estimate, for each of the plurality of pixel locations using the series of images, a single brightness value between consecutive changepoints; and
generate the first image based on the estimated single brightness value for each of the plurality of pixel locations at the first point in the series of images.

5. The system of claim 1 , wherein the at least one processor is further programmed to:
determine motion between at least a portion of the first image and at least a portion of the second image.

6. The system of claim 5 , wherein the at least one processor is further programmed to:
generate a deblurred image based on the motion between the first image and the second image and pixel values of each of the series of images.

7. The system of claim 5 , wherein the at least one processor is further programmed to:
generate an upsampled image based on the motion between the first image and the second image and pixel values of each of the series of images.

8. The system of claim 1 , wherein the at least one processor is further programmed to:
provide the first image and the second image to a machine learning model.

9. The system of claim 1 , wherein the threshold portion of the plurality of pixel locations is at least one percent of the plurality of pixel locations.

10. A method for generating digital images, comprising:
receiving, from an image sensor comprising a plurality of detectors configured to detect arrival of photons, a series of images, each image in the series of images comprising a plurality of pixels representing brightness of a portion of a scene, each of the plurality of pixels associated with a pixel location (x,y);
identifying, for each of the plurality of pixel locations using the series of images, one or more changepoints at which a transition in brightness associated with a pixel between different steady state brightness values occurs;
determining, based on the one or more changepoints associated with the plurality of pixel locations, a first point in the series of images at which at least a threshold portion of plurality of pixel locations have included at least one changepoint;
generating a first image based on brightness associated with each pixel location at the first point in the series of images;
determine, based on changepoints associated with the plurality of pixel locations, a second point in the series of images at which at least the threshold portion of the plurality of pixel locations have included at least one changepoint that occurred after the first point in the series of images; and
generate a second image based on brightness associated with each pixel location at the second point in the series of images.

11. The method of claim 10 , wherein each of the plurality of detectors is a single-photon avalanche diode (SPAD).

12. The method of claim 11 , wherein each SPAD is configured to record a timestamp at which a single photon was detected.

13. The method of claim 10 , further comprising:
estimating, for each of the plurality of pixel locations using the series of images, a single brightness value between consecutive changepoints; and
generating the first image based on the estimated single brightness value for each of the plurality of pixel locations at the first point in the series of images.

14. The method of claim 10 , further comprising:
determining motion between the first image and the second image.

15. The method of claim 14 , further comprising:
generating a deblurred image based on the motion between the first image and the second image and pixel values of each of the series of images.

16. The method of claim 14 , further comprising:
generating an upsampled image based on the motion between the first image and the second image and pixel values of each of the series of images.

17. The method of claim 10 , further comprising:
providing the first image and the second image to a machine learning model.

18. The method of claim 10 , wherein the threshold portion of the plurality of pixel locations is at least one percent of the plurality of pixel locations.

19. A system for generating digital images, comprising:
an image sensor comprising a plurality of single-photon detectors configured to detect arrival of photons, the plurality of single-photon detectors arranged in an array;
at least one processor that is programmed to:
receive a sequence of photon frames, each photon frame comprising a plurality of pixels having a value indicative of whether a photon was received during an associated frame period, wherein each of the plurality of pixels corresponds to a pixel location of a plurality of pixel locations;
identify, for each of the plurality of pixel locations, a plurality of changepoints, each of the plurality of changepoints is indicative of a change in scene brightness;
identify a photon frame in the sequence of photon frames at which at least a threshold change in brightness has occurred based on the changepoints associated with each of the plurality of pixel locations; and
generate a series of changepoint frames, wherein each changepoint frame is based on estimated brightness associated with each pixel location at a point in the sequence of photon frames.

20. The system of claim 19 , wherein the at least one processor that is further programmed to:
identify each of plurality of changepoints using no information from photon frames following a photon frame corresponding to the respective changepoint in the sequence.

Assignees

• Wisconsin Alumni Res Found

Inventors

• Seets Trevor
• Ingle Atul
• Velten Andreas

Classifications

• H04N23/71 Primary

  Circuitry for evaluating the brightness variation · CPC title

• H04N5/2351 Primary

  Electricity · mapped topic

• H04N25/773

  comprising photon counting circuits, e.g. single photon detection [SPD] or single photon avalanche diodes [SPAD] · CPC title

• H04N23/683

  performed by a processor, e.g. controlling the readout of an image memory · CPC title

• H04N23/6811

  based on the image signal · CPC title