Method and system for digital staining of label-free fluorescence images using deep learning

Patent metadata

Field	Value
Publication number	US-11893739-B2
Application number	US-201917041447-A
Country	US
Kind code	B2
Filing date	Mar 29, 2019
Priority date	Mar 30, 2018
Publication date	Feb 6, 2024
Grant date	Feb 6, 2024

Abstract

A deep learning-based digital staining method and system are disclosed that enables the creation of digitally/virtually-stained microscopic images from label or stain-free samples based on autofluorescence images acquired using a fluorescent microscope. The system and method have particular applicability for the creation of digitally/virtually-stained whole slide images (WSIs) of unlabeled/unstained tissue samples that are analyzes by a histopathologist. The methods bypass the standard histochemical staining process, saving time and cost. This method is based on deep learning, and uses, in one embodiment, a convolutional neural network trained using a generative adversarial network model to transform fluorescence images of an unlabeled sample into an image that is equivalent to the brightfield image of the chemically stained-version of the same sample. This label-free digital staining method eliminates cumbersome and costly histochemical staining procedures and significantly simplifies tissue preparation in pathology and histology fields.

First claim

What is claimed is:

1. A method of generating a digitally stained microscopic image of a label-free sample comprising:
providing a trained, deep neural network that is executed by image processing software using one or more processors of a computing device, wherein the trained, deep neural network is trained with a plurality of matched chemically stained images or image patches and their corresponding fluorescence images or image patches of the same sample;
obtaining one or more fluorescence images of the sample using a fluorescence microscope and one or more excitation light sources, wherein fluorescent light is emitted from endogenous fluorophores or other endogenous emitters of frequency-shifted light within the sample;
inputting the one or more fluorescence images of the sample to the trained, deep neural network; and
the trained, deep neural network outputting the digitally stained microscopic image of the sample that is substantially equivalent to a corresponding brightfield image of the same sample that has been chemically stained.

2. The method of claim 1 , wherein the trained, deep neural network comprises a convolutional neural network.

3. The method of claim 1 , wherein the deep neural network is trained using a Generative Adversarial Network (GAN) model.

4. The method of claim 1 , wherein the sample is further labeled with one or more exogenous fluorescent labels or other exogenous emitters of light.

5. The method of claim 1 , wherein the deep neural network is trained using a generator network configured to learn statistical transformation between the matched chemically stained and fluorescence images or image patches of the same sample and a discriminator network configured to discriminate between a ground truth chemically stained image of the sample and the outputted digitally stained microscopic image of the sample.

6. The method of claim 1 , wherein the sample comprises mammalian tissue, plant tissue, cells, pathogens, biological fluid smears, or other objects of interest.

7. The method of claim 1 , wherein the deep neural network is trained with samples of the same type as the sample type of the obtained one or more fluorescence image(s).

8. The method of claim 1 , wherein the trained, deep neural network outputs a digitally stained microscopic image in less than one second of inputting the one or more fluorescence image(s).

9. The method of claim 1 , wherein the sample comprises a non-fixed tissue sample.

10. The method of claim 1 , wherein the sample comprises a fixed tissue sample.

11. The method of claim 10 , wherein the fixed tissue sample is embedded in paraffin.

12. The method of claim 1 , wherein the sample comprises a fresh tissue sample.

13. The method of claim 1 , wherein the sample comprises tissue imaged in vivo.

14. The method of claim 1 , wherein the excitation light source emits ultra-violet or near ultra-violet light.

15. The method of claim 1 , wherein the one or more fluorescence images are obtained at a filtered emission band or emission wavelength range using one or more filters of a filter set.

16. The method of claim 15 , wherein a plurality of filters are used to capture a plurality of fluorescence images which are input to the trained, deep neural network.

17. The method of claim 16 , wherein the plurality of fluorescence images are obtained by multiple excitation light sources emitting light at different wavelengths or wavelength bands.

18. The method of claim 1 , wherein the one or more fluorescence images is/are subject to one or more linear or non-linear image pre-processing operations selected from contrast enhancement, contrast reversal, image filtering prior to being input to the trained, deep neural network.

19. The method of claim 17 , wherein the one or more fluorescence images and one or more pre-processed images are input together into the trained, deep neural network.

20. The method of claim 1 , wherein the plurality of matched chemically stained and fluorescence images or image patches of the same sample are subject to registration during training, comprising a global registration process that corrects for rotation and a subsequent local registration process that matches local features found in the matched chemically stained and fluorescence images.

21. The method of claim 1 , wherein the trained, deep neural network is trained using one or more GPUs or ASICs.

22. The method of claim 1 , wherein the trained, deep neural network is executed using one or more GPUs or ASICs.

23. The method of claim 1 , wherein the digitally stained microscopic image of the sample is output in real time or near real time after obtaining the one or more fluorescence images of the sample.

24. The method of claim 23 , wherein the trained, deep neural network is trained for a second tissue/stain combination using initial neural network weights and biases from a first tissue/stain combination which are optimized for the second tissue/stain combination using transfer learning.

25. The method of claim 23 , wherein the trained, deep neural network is trained for multiple tissue/stain combinations.

26. The method of claim 23 , wherein the trained, deep neural network is trained for more than one chemical stain type for a given tissue type.

27. A method of generating a digitally stained microscopic image of a label-free sample comprising:
providing a trained, deep neural network that is executed by image processing software using one or more processors of a computing device, wherein the trained, deep neural network is trained with a plurality of matched chemically stained images or image patches and their corresponding fluorescence images or image patches of the same sample;
obtaining a first fluorescence image of the sample using a fluorescence microscope and wherein fluorescent light at a first wavelength or wavelength range is emitted from endogenous fluorophores or other endogenous emitters of frequency-shifted light within the sample;
obtaining a second fluorescence image of the sample using a fluorescence microscope and wherein fluorescent light at a second wavelength or wavelength range is emitted from endogenous fluorophores or other endogenous emitters of frequency-shifted light within the sample
inputting the first and second fluorescence images of the sample to the trained, deep neural network; and
the trained, deep neural network outputting the digitally stained microscopic image of the sample that is substantially equivalent to a corresponding brightfield image of the same sample that has been chemically stained.

28. The method of claim 27 , wherein the first fluorescence image and the second fluorescence image are obtained using different resolutions.

29. The method of claim 27 , wherein the sample comprises tissue, cells, pathogens, biological fluid smears, or other objects of interest.

30. A system for generating digitally stained microscopic images of a chemically unstained sample comprising:
a computing device having image processing software executed thereon or thereby, the image processing software comprising a trained, deep neural network that is executed using one or more processors of the computing device, wherein the trained, deep neural network is trained with a plurality of matched chemically stained images or image patches and their corresponding fluorescence images

Assignees

• Univ California

Inventors

• Ozcan Aydogan
• Rivenson Yair
• Wang Hongda
• Wei Zhensong

Classifications

• G06N3/0475

  Generative networks · CPC title

• G06N3/094

  Adversarial learning · CPC title

• G06N3/096

  Transfer learning · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/09

  Supervised learning · CPC title