Processing of histology images with a convolutional neural network to identify tumors

Patent metadata

Field	Value
Publication number	US-11449998-B2
Application number	US-202016986987-A
Country	US
Kind code	B2
Filing date	Aug 6, 2020
Priority date	Dec 29, 2017
Publication date	Sep 20, 2022
Grant date	Sep 20, 2022

Abstract

A convolutional neural network (CNN) is applied to identifying tumors in a histological image. The CNN has one channel assigned to each of a plurality of tissue classes that are to be identified, there being at least one class for each of non-tumorous and tumorous tissue types. Multi-stage convolution is performed on image patches extracted from the histological image followed by multi-stage transpose convolution to recover a layer matched in size to the input image patch. The output image patch thus has a one-to-one pixel-to-pixel correspondence with the input image patch such that each pixel in the output image patch has assigned to it one of the multiple available classes. The output image patches are then assembled into a probability map that can be co-rendered with the histological image either alongside it or over it as an overlay. The probability map can then be stored linked to the histological image.

First claim

What is claimed is:

1. A method comprising using at least one hardware processor to, for each of one or more histological images of stained tissue samples:
receive the histological image;
extract a plurality of image patches from the histological image, wherein each of the plurality of image patches comprises a partial region of the histological image;
apply a convolutional neural network (CNN) to the plurality of image patches to:
classify each pixel of each of the plurality of image patches into one of a plurality of classes, wherein the plurality of classes comprises a non-tumor class representing an absence of tumor tissue and at least one tumor class representing tumor tissue, and
generate a plurality of output image patches based on classifying each pixel of each of the plurality of image patches into one of the plurality of classes, wherein a first pixel of an output image patch of the plurality of output image patches is classified into a first class of the plurality of classes and a second pixel of the output image patch is classified into a second class of the plurality of classes; and

generate a probability heatmap from the plurality of output image patches, wherein the probability heatmap is overlaid over the histological image, wherein the probability heatmap assigns a color to each pixel in the histological image based on the one class into which that pixel was classified.

2. The method of claim 1 , further comprising using the at least one hardware processor to train the CNN using training data comprising histological images that have been annotated according to the plurality of classes.

3. The method of claim 2 , wherein the training uses gradient descent.

4. The method of claim 1 , wherein the at least one tumor class comprises a plurality of tumor classes, wherein each of the plurality of tumor classes represents a different type of tumor.

5. The method of claim 4 , wherein the plurality of tumor classes comprises a first tumor class representing an invasive tumor and a second tumor class representing an in situ tumor.

6. The method of claim 1 , wherein to classify each pixel of each of the plurality of image patches into one of the plurality of classes, the method further comprises using the at least one hardware processor to, for each pixel:
output, by the CNN, a vector of probabilities, wherein each probability in the vector is a probability that the pixel belongs to one of the plurality of classes; and
select a class of the plurality of classes into which the pixel is classified based on the vector of probabilities.

7. The method of claim 1 , wherein the CNN comprises a plurality of convolutional layers and a plurality of deconvolutional layers.

8. The method of claim 7 , wherein the CNN comprises a maxpool layer following at least one of the plurality of convolutional layers.

9. The method of claim 8 , wherein the CNN comprises a skip connection between at least one maxpool layer and one of the plurality of deconvolutional layers.

10. The method of claim 9 , wherein, for each of the plurality of image patches, the plurality of deconvolutional layers upscales a convolutional feature map of the image patch to a feature map with a same width and height as the image patch.

11. The method of claim 10 , wherein each of the plurality of deconvolutional layers enlarges a width and height of an input feature map by a factor of two.

12. The method of claim 7 , wherein to apply CNN, the method further comprises using the at least one hardware processor to, for each of the plurality of image patches:
concatenate two or more features maps, produced by different ones of the plurality of deconvolutional layers, into a combined feature map; and
convert the combined feature map into a probability map using a softmax operation, the probability map comprising a vector of probabilities for each pixel in the image patch, wherein each probability in each vector is a probability that the pixel belongs to one of the plurality of classes.

13. The method of claim 7 , wherein an output of one or more of the plurality of convolutional layers or the plurality of deconvolutional layers is transformed by a rectifier function.

14. The method of claim 1 , wherein none of the plurality of images patches overlap.

15. The method of claim 1 , wherein two or more of the plurality of image patches overlap.

16. The method of claim 1 , wherein to generate the probability heatmap, the method further comprises using the at least one hardware processor to, stitch the plurality of output image patches together, wherein each of the plurality of output image patches represents a probability map for one of the plurality of image patches.

17. The method of claim 1 , wherein the plurality of image patches covers an entirety of the histological image.

18. The method of claim 1 , wherein the plurality of image patches only covers a portion of the histological image.

19. A system comprising:
at least one hardware processor; and
one or more software modules that are configured to, when executed by the at least one hardware processor, for each of one or more histological images of stained tissue samples,
receive the histological image,
extract a plurality of image patches from the histological image, wherein each of the plurality of image patches comprises a partial region of the histological image;
apply a convolutional neural network (CNN) to the plurality of image patches to:
classify each pixel of each of the plurality of image patches into one of a plurality of classes, wherein the plurality of classes comprises a non-tumor class representing an absence of tumor tissue and at least one tumor class representing tumor tissue, and
generate a plurality of output image patches based on classifying each pixel of each of the plurality of image patches into one of the plurality of classes, wherein a first pixel of an output image patch of the plurality of output image patches is classified into a first class of the plurality of classes and a second pixel of the output image patch is classified into a second class of the plurality of classes; and

generate a probability heatmap from the plurality of output image patches, wherein the probability heatmap is overlaid over the histological image, wherein the probability heatmap assigns a color to each pixel in the histological image based on the one class into which that pixel was classified.

20. A non-transitory computer-readable medium having instructions stored thereon, wherein the instructions, when executed by a processor, cause the processor to, for each of one or more histological images of stained tissue samples:
receive the histological image;
extract a plurality of image patches from the histological image, wherein each of the plurality of image patches comprises a partial region of the histological image;
apply a convolutional neural network (CNN) to the plurality of image patches to:
classify each pixel of each of the plurality of image patches into one of a plurality of classes, wherein the plurality of classes comprises a non-tumor class representing an absence of tumor tissue and at least one tumor class representing tumor tissue, and
generate a plurality of output image patches based on classifying each pixel of each of the plurality of image patches into one of the plurality of classes, wherein a first pixel of an output image patch of the plurality of output image patches is classified into a first class of the plurality of classes and a second pixel of the output ima

Assignees

• Leica Biosystems Imaging Inc

Inventors

• Georgescu Walter
• Olson Allen
• Annaldas Bharat
• Lawler Darragh
• Shields Kevin
• Saligrama Kiran
• Gregson Mark

Classifications

• G16H30/40 Primary

  for processing medical images, e.g. editing · CPC title

• G06V30/19173

  Classification techniques · CPC title

• G06V10/82

  using neural networks · CPC title

• G06F2218/12

  Classification; Matching · CPC title

• G06F2218/08

  Feature extraction · CPC title