Systems and methods for deep learning microscopy

Patent metadata

Field	Value
Publication number	US-11222415-B2
Application number	US-201916395674-A
Country	US
Kind code	B2
Filing date	Apr 26, 2019
Priority date	Apr 26, 2018
Publication date	Jan 11, 2022
Grant date	Jan 11, 2022

Abstract

A microscopy method includes a trained deep neural network that is executed by software using one or more processors of a computing device, the trained deep neural network trained with a training set of images comprising co-registered pairs of high-resolution microscopy images or image patches of a sample and their corresponding low-resolution microscopy images or image patches of the same sample. A microscopy input image of a sample to be imaged is input to the trained deep neural network which rapidly outputs an output image of the sample, the output image having improved one or more of spatial resolution, depth-of-field, signal-to-noise ratio, and/or image contrast.

First claim

What is claimed is:

1. A microscopy method comprising:
providing a trained deep neural network that is executed by software using one or more processors of a computing device, the trained deep neural network trained with a training set of non-fluorescence images comprising co-registered pairs of high-resolution microscopy images or image patches of a sample and their corresponding low-resolution microscopy images or image patches of the same sample;
inputting a non-fluorescence microscopy input image of a second sample to the trained deep neural network;
outputting an output image of the second sample from the trained deep neural network, the output image having improved one or more of spatial resolution, depth-of-field, signal-to-noise ratio, and/or image contrast; and
wherein the non-fluorescence microscopy input image comprises one of a bright-field microscopy image, a holographic microscopy image, a dark-field microscopy image, and an optical coherence tomography (OCT) image.

2. The microscopy method of claim 1 , wherein the trained deep neural network comprises a trained convolutional neural network (CNN).

3. The method of claim 2 , wherein the trained CNN is trained using a training set of non-fluorescence images and wherein a parameter space of the CNN is established during the training by optimization of a statistical transformation between the low-resolution microscopy images or image patches and the high-resolution microscopy images or image patches.

4. The method of claim 3 , wherein the trained CNN is trained as a generative adversarial network (GAN) model comprising first and second sub-networks trained simultaneously, wherein the first sub-network comprises a generative model configured to enhance the input low-resolution images or image patches and the second sub-network comprises a discriminative model configured to return an adversarial loss to a resolution-enhanced image or image patch, resulting from the generative model.

5. A microscopy method comprising:
providing a trained deep neural network that is executed by software using one or more processors of a computing device, the trained deep neural network trained with a training set of non-fluorescence histopathological slide images of tissue comprising co-registered pairs of high-resolution microscopy images or image patches of a tissue sample and their corresponding low-resolution microscopy images or image patches of the same tissue sample;
inputting a non-fluorescence microscopy input histopathological slide image of a second sample of tissue to the trained deep neural network; and
outputting an output image of the second sample of tissue from the trained deep neural network, the output image having improved one or more of spatial resolution, depth-of-field, signal-to-noise ratio, and/or image contrast.

6. The microscopy method of claim 5 , wherein the high-resolution microscopy images or image patches are obtained by synthesizing a higher resolution image from multiple, sub-pixel shifted low-resolution images.

7. The microscopy method of claim 5 , wherein the trained deep neural network is trained using a training set of non-fluorescence images of tissue of the same type of tissue as the second sample.

8. The microscopy method of claim 5 , wherein the trained deep neural network is trained using a training set of non-fluorescence images of tissue of a different type of tissue as the second sample.

9. The microscopy method of claim 5 , wherein the trained deep neural network is trained using a training set of non-fluorescence images of tissue stained with the same stain or dye used to stain the second sample.

10. The microscopy method of claim 5 , wherein the trained deep neural network is trained using a training set of non-fluorescence images of tissue stained with a different stain or dye used to stain the second sample.

11. A microscopy method comprising:
providing a trained deep neural network that is executed by software using one or more processors of a computing device, the trained deep neural network trained with a training set of non-fluorescence images comprising co-registered pairs of high-resolution microscopy images or image patches of a sample and their corresponding low-resolution microscopy images or image patches of the same sample;
inputting a non-fluorescence microscopy input image of a second sample to the trained deep neural network;
outputting an output image of the second sample from the trained deep neural network, the output image having improved one or more of spatial resolution, depth-of-field, signal-to-noise ratio, and/or image contrast; and
wherein the high-resolution microscopy images or image patches are obtained by synthesizing a higher resolution image from multiple, sub-pixel shifted low-resolution images.

12. A microscopy method comprising:
providing a trained deep neural network that is executed by software using one or more processors of a computing device, the trained deep neural network trained with a training set of fluorescence images comprising co-registered pairs of high-resolution microscopy images or image patches of one or more samples and their corresponding low-resolution microscopy images or image patches of the same sample(s), wherein each one of the high-resolution microscopy images or image patches of the sample(s) and their corresponding low-resolution microscopy images or image patches comprises an image that captures at a single image exposure a fluorescence radiation signal emitted from the entire sample that lies within the image;
inputting a fluorescence microscopy input image of a second sample to the trained deep neural network that comprises an image that captures at a single image exposure a fluorescence radiation signal emitted from the entire sample that lies within the image;
outputting an output image of the second sample from the trained deep neural network, the output image having improved one or more of spatial resolution and depth-of-field, signal-to-noise ratio, and/or image contrast.

13. The microscopy method of claim 12 , wherein the training set of images comprise histopathological slide images of tissue and the microscopy input image comprises a histopathological slide image of tissue.

14. The microscopy method of claim 13 , wherein the trained deep neural network is trained using a training set of images of tissue of the same type of tissue as the second sample.

15. The microscopy method of claim 13 , wherein the trained deep neural network is trained using a training set of images of tissue of a different type of tissue as the second sample.

16. The microscopy method of claim 13 , wherein the trained deep neural network is trained using a training set of images of tissue stained with the same stain or dye used to stain the second sample.

17. The microscopy method of claim 13 , wherein the trained deep neural network is trained using a training set of images of tissue stained with a different stain or dye used to stain the second sample.

18. The microscopy method of claim 12 , wherein the fluorescence microscopy input image comprises a multi-photon microscopy image and/or a confocal microscopy image.

19. The microscopy method of claim 12 , wherein the trained deep neural network comprises a trained convolutional neural network (CNN).

20. The microscopy method of claim 12 , wherein the output image of the second sample has spatial frequency spectra that substantially matches that obtained from a higher-resolution image of the same field-of-view.

Assignees

• Univ California

Inventors

• Ozcan Aydogan
• Rivenson Yair
• Wang Hongda
• Gunaydin Harun
• De Haan Kevin

Classifications

• G06N3/048

  Activation functions · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G06N3/047

  Probabilistic or stochastic networks · CPC title

• G06N3/0475

  Generative networks · CPC title

• G06N3/09

  Supervised learning · CPC title