Fluorescence lifetime imaging using deep learning

Patent metadata

Field	Value
Publication number	US-12266104-B2
Application number	US-202418423472-A
Country	US
Kind code	B2
Filing date	Jan 26, 2024
Priority date	Jan 7, 2020
Publication date	Apr 1, 2025
Grant date	Apr 1, 2025

Abstract

One embodiment provides an apparatus for fluorescence lifetime imaging (FLI). The apparatus includes a deep neural network (DNN). The DNN includes a first convolutional layer, a plurality of intermediate layers and an output layer. The first convolutional layer is configured to receive FLI input data. Each intermediate layer is configured to receive a respective intermediate input corresponding to an output of a respective prior layer. Each intermediate layer is further configured to provide a respective intermediate output related to the received respective intermediate input. The output layer is configured to provide estimated FLI output data corresponding to the received FLI input data. The DNN is trained using synthetic data.

First claim

What is claimed is:

1. An apparatus for fluorescence lifetime imaging (FLI), the apparatus comprising:
a deep neural network (DNN) comprising:
a first convolutional layer configured to receive FLI input data;
a plurality of intermediate layers, each intermediate layer configured to receive a respective intermediate input corresponding to an output of a respective prior layer, each intermediate layer further configured to provide a respective intermediate output related to the received respective intermediate input; and
an output layer configured to provide estimated FLI output data corresponding to the received FLI input data;

wherein the DNN is trained using synthetic data; and
a discriminator network configured to compare estimated training output data with training synthetic output data during training, the DNN and the discriminator network corresponding to a generative adversarial network (GAN) during training, the discriminator network comprising:
a first two-dimensional (2D) convolutional block;
a first intermediate block coupled to an output of the first 2D convolutional block;
a second 2D convolutional block coupled to an output of the first intermediate block;
a second intermediate block coupled to an output of the second 2D convolutional block;
a flatten block coupled to an output of the second intermediate block;
a first dense block coupled to an output of the flatten block;
a third intermediate block coupled to an output of the first dense block;
a second dense block coupled to an output of the third intermediate block;
a dropout block coupled to an output of the second dense block; and
a sigmoid block coupled to an output of the dropout block;
wherein the first intermediate block, the second intermediate block, and the third intermediate block each comprise a batch normalization function and a rectified linear unit.

2. The apparatus of claim 1 , wherein the first convolutional layer is a three-dimensional (3D) convolutional layer, the plurality of intermediate layers comprises a 3D residual block, a reshape layer, a two-dimensional (2D) convolutional layer, and at least one 2D residual block, and the output layer is a fully convolutional (FC) down-sample layer.

3. The apparatus of claim 1 , wherein the first convolutional layer is a separable two-dimensional (2D) convolutional layer; the plurality of intermediate layers comprises a 2D exception block, a 2D convolutional layer, and a 2D residual block; and the output layer is a fully convolutional (FC) down-sample layer.

4. The apparatus of claim 3 , wherein the 2D exception block comprises a first 2D separable convolution block, a batch normalization and rectified linear unit block, a second 2D separable convolution block, a batch normalization block, an adder block configured to add an output from the batch normalization block and an input of the 2D exception block that corresponds to an output from the separable 2D convolutional layer, and a rectified linear unit block.

5. The apparatus of claim 1 , wherein the DNN further comprises a second convolutional layer configured to receive optical property data, the plurality of intermediate layers comprises a concatenate layer, and the estimated FLI output data is further related to the optical property data.

6. The apparatus of claim 1 , wherein the first convolutional layer is a three-dimensional (3D) convolutional layer, the plurality of intermediate layers comprises a second 3D convolutional layer, a reshape layer, a separable two-dimensional (2D) convolutional layer, and a 2D exception block, and the DNN comprises a plurality of output layers and each output layer is a coefficient block that corresponds to a fully convolutional (FC) down-sample layer.

7. The apparatus of claim 1 , wherein the FLI input data is selected from the group comprising visible FLI microscopy (FLIM) data, near infrared (NIR) FLIM data, and NIR gated macroscopy FLI (MFLI) data.

8. A method for fluorescence lifetime imaging (FLI), the method comprising:
receiving, by a first convolutional layer of a deep neural network (DNN), FLI input data;
receiving, by each intermediate layer of a plurality of intermediate layers, a respective intermediate input corresponding to an output of a respective prior layer;
providing, by each intermediate layer, a respective intermediate output related to the received respective intermediate input;
providing, by an output layer, estimated FLI output data corresponding to the received FLI input data;
wherein the DNN is trained using synthetic data; and
comparing, by a discriminator network, estimated training output data with training synthetic output data during training, the DNN and the discriminator network corresponding to a generative adversarial network (GAN) during training; the discriminator network comprising:
a first two-dimensional (2D) convolutional block;
a first intermediate block coupled to an output of the first 2D convolutional block;
a second 2D convolutional block coupled to an output of the first intermediate block;
a second intermediate block coupled to an output of the second 2D convolutional block;
a flatten block coupled to an output of the second intermediate block;
a first dense block coupled to an output of the flatten block;
a third intermediate block coupled to an output of the first dense block;
a second dense block coupled to an output of the third intermediate block;
a dropout block coupled to an output of the second dense block; and
a sigmoid block coupled to an output of the dropout block;
wherein the first intermediate block, the second intermediate block, and the third intermediate block each comprise a batch normalization function and a rectified linear unit.

9. The method of claim 8 , wherein the first convolutional layer is a three-dimensional (3D) convolutional layer, the plurality of intermediate layers comprises a 3D residual block, a reshape layer, a two-dimensional (2D) convolutional layer, and at least one 2D residual block, and the output layer is a fully convolutional (FC) down-sample layer.

10. The method of claim 8 , wherein the first convolutional layer is a separable two-dimensional (2D) convolutional layer; the plurality of intermediate layers comprises a 2D exception block, a 2D convolutional layer, and a 2D residual block; and the output layer is a fully convolutional (FC) down-sample layer.

11. The method of claim 10 , wherein the 2D exception block comprises a first 2D separable convolution block, a batch normalization and rectified linear unit block, a second 2D separable convolution block, a batch normalization block, an adder block configured to add an output from the batch normalization block and an input of the 2D exception block that corresponds to an output from the separable 2D convolutional layer, and a rectified linear unit block.

12. The method of claim 8 , further comprising receiving, by a second convolutional layer, optical property data, the plurality of intermediate layers comprising a concatenate layer, and the estimated FLI output data is further related to the optical property data.

13. The method of claim 8 , wherein the first convolutional layer is a three-dimensional (3D) convolutional layer, the plurality of intermediate layers comprises a second 3D convolutional layer, a reshape layer, a separable two-dimensional (2D) convolutional layer, and a 2D exception block, and the DNN comprises a plurality of output layers and each output layer is a coefficient block that corresponds to a fully convolutional (FC) down-sample layer.

14. The method of claim 8 , wherein the FLI input data is selected from the group

Assignees

• Rensselaer Polytech Inst

Inventors

• Smith Jason Tyler
• Yao Ruoyang
• Intes Xavier
• Yan Pingkun
• Ochoa-Mendoza Marien

Classifications

• G06N3/0475

  Generative networks · CPC title

• G06N3/094

  Adversarial learning · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/045

  Combinations of networks · CPC title