Low-light video system

Patent metadata

Field	Value
Publication number	US-12243195-B2
Application number	US-202217661338-A
Country	US
Kind code	B2
Filing date	Apr 29, 2022
Priority date	Apr 29, 2022
Publication date	Mar 4, 2025
Grant date	Mar 4, 2025

Abstract

Real time, low-light images, for example, obtained from the fluorescent marker for identifying tumors during surgery, are combined to improve the signal-to-noise ratio using a motion signal derived from corresponding high-light images, for example, taken with a second camera at interleaved intervals of higher illumination.

First claim

We claim:

1. A low-light video system comprising:
at least one camera adapted to:
receive low light from an imaged object to provide a sequence of low-light image frames;
receive high light from the imaged object having a greater flux than the low light to provide a sequence of high-light image frames;
an electronic processor implementing:
(a) a motion extractor receiving the high-light image frames from the at least one camera to determine motion of the imaged object between high-light image frames; and
(b) an integrator combining low-light image frames after alignment according to the motion determined by the motion extractor to output reduced noise low-light image frames; and
further including a neural network receiving the reduced noise low-light image frames and outputting corrected low-light image frames, the neural network trained with a teaching set of pairs of low-light image frames with respectively higher and lower levels of noise with respect to a common imaged object.

2. The low-light video system of claim 1 wherein the higher and lower levels of noise are differences selected from the group consisting of random additive noise, spatial distortion, blurring, and quantization noise.

3. The low-light video system of claim 1 wherein the teaching set of low-light image frames are of tissue.

4. The low-light video system of claim 3 wherein each teaching set pair of low-light image frames include a fluorescence image of tissue and the same fluorescence image of tissue with added simulated noise.

5. The low-light video system of claim 3 wherein the teaching set of low-light image frames represents images taken with the at least one camera of the tissue and wherein the teaching set further includes high-light image frames representing images taken with the at least one camera of the tissue and where the neural network further receives the high-light image data.

6. The low-light video system of claim 1 further including an error detector producing an error signal indicating errors in the determined motion relating to at least a portion of a high-light image frame and wherein the integrator uses the error signal to exclude a portion of a corresponding low-light image frame from the combining.

7. The low-light video system of claim 6 wherein the error signal is produced by warping an early received high-light image frame according to the motion with respect to a later received-light image frame and comparing the warped early received high-light image frame to the later received high-light image frame to identify pixels having differences in value of more than a predefined threshold, the determined pixels providing the error signal.

8. The low-light video system of claim 1 wherein the integrator combines different numbers of low-light image frames for different pixels of the low-light image frames.

9. The low-light video system of claim 1 further including an error detector producing an error signal indicating errors in the determined motion and wherein the integrator combines different numbers of low-light image frames for different given pixels of the low-light image frames according to a number of low-light image frames occurring after an error signal including the given pixel.

10. The low-light video system of claim 1 further including a synchronization circuit synchronizing an acquisition of the sequence of low-light image frames and sequence of high-light image frames with an area illuminator switching between an on-state and off-state so that the low-light image frames are obtained only during the on-state and high-light image frames are obtained only during the off-state.

11. The low-light video system of claim 1 wherein the at least one camera includes a filter selectively passing infrared light and blocking visible light.

12. The low-light video system of claim 1 wherein the low-light image frame have a lower image resolution than the high-light image frames.

13. The low-light video system of claim 1 wherein the at least one camera is a single photon camera.

14. A method of low-light imaging using a system including:
at least one camera adapted to:
receive low light from an imaged object to provide a sequence of low-light image frames;
receive high light from the imaged object having a greater flux than the low light to provide a sequence of high-light image frames; and
an electronic processor implementing:
a motion extractor receiving the high-light image frames from the at least one camera to determine motion of the imaged object between high-light image frames;
an integrator combining low-light image frames after alignment according to the motion determined by the motion extractor to output an image based on combined low-light image frames, and
a neural network receiving the reduced noise low-light image frames and outputting corrected low-light image frames, the neural network trained with a teaching set of pairs of low-light image frames with respectively higher and lower levels of noise with respect to a common imaged object;
the method comprising:
(a) obtaining a sequence of low-light image frames and a corresponding sequence of high-light image frames of an object subject to motion;
(b) using the high-light image frames to deduce motion of the subject; and
(c) combining the low-light image frames after alignment according to the motion deduced from the high-light image frames; and
(d) processing the reduced noise low-light image frames using the neural network and outputting corrected low-light image frames.

15. A low-light video system comprising:
at least one camera adapted to:
receive low light from an imaged object to provide a sequence of low-light image frames;
receive high light from the imaged object having a greater flux than the low light to provide a sequence of high-light image frames; and
a neural network receiving data from the low-light image frames and the high-light image frames to output an image based on a combination of the data of the low-light image frames and the high-light image frames, the neural network trained with a teaching set of pairs of low-light image frames with respectively higher and lower levels of noise with respect to a common imaged object.

16. The low-light video system of claim 15 wherein the higher and lower levels of noise are differences selected from the group consisting of random additive noise, spatial distortion, blurring, and quantization noise.

17. The low-light video system of claim 15 wherein the teaching set of low-light image frames are of tissue.

18. The low-light video system of claim 17 wherein each teaching set pair of low-light image frames includes a fluorescence image of tissue and the same fluorescence image of tissue with added simulated noise.

19. The low-light video system of claim 17 wherein the teaching set of low-light image frames represents images taken with the at least one camera of the tissue and wherein the teaching set further includes high-light image frames representing images taken with the at least one camera of the tissue.

20. The low-light video system of claim 15 wherein the low-light image frame have a lower image resolution than the high-light image frames.

21. The low-light video system of claim 15 wherein the at least one camera is a single photon camera.

Assignees

• Wisconsin Alumni Res Found
• Onlume Inc

Inventors

• Velten Andreas
• Seets Trevor
• Lin Wei
• Lin Christie
• Uselmann Adam
• Lu Yizou

Classifications

• G06T2207/20182

  Noise reduction or smoothing in the temporal domain; Spatio-temporal filtering · CPC title

• G06T2207/10068

  Endoscopic image · CPC title

• G06T2207/10016

  Video; Image sequence · CPC title

• G06T5/70 Primary

  Denoising; Smoothing · CPC title

• G06T2207/20221

  Image fusion; Image merging · CPC title