Neural network based identification of areas of interest in digital pathology images

Patent metadata

Field	Value
Publication number	US-12094182-B2
Application number	US-202017416394-A
Country	US
Kind code	B2
Filing date	May 29, 2020
Priority date	May 29, 2019
Publication date	Sep 17, 2024
Grant date	Sep 17, 2024

Abstract

A CNN is applied to a histological image to identify areas of interest. The CNN classifies pixels according to relevance classes including one or more classes indicating levels of interest and at least one class indicating lack of interest. The CNN is trained on a training data set including data which has recorded how pathologists have interacted with visualizations of histological images. In the trained CNN, the in-terest-based pixel classification is used to generate a segmentation mask that defines areas of interest. The mask can be used to indicate where in an image clinically relevant features may be located. Further, it can be used to guide variable data compression of the histological image. Moreover, it can be used to control loading of image data in either a client-server model or within a memory cache policy. Furthermore, a histological image of a tissue sample of a tissue type that has been treated with a test compound is image processed in order to detect areas where toxic reactions to the test compound may have occurred. An autoencoder is trained with a training data set comprising histological images of tissue samples which are of the given tissue type, but which have not been treated with the test compound. The trained autoencoder is applied to detect tissue areas by their deviation from the normal variation seen in that tissue type as learnt by the training process, and so build up a toxicity map of the image. The toxicity map can then be used to direct a toxicological pathologist to examine the areas identified by the autoencoder as lying outside the normal range of heterogeneity for the tissue type. This makes the pathologist's review quicker and more reliable. The toxicity map can also be overlayed with the segmentation mask indicating areas of interest. When an area of interest and an area identified as lying outside the normal range of heterogeneity for the tissue type, and increased confidence score is applied to the overlapping area.

First claim

What is claimed is:

1. An apparatus comprising:
a memory configured to store computer-executable instructions; and
a hardware processor in communication with the memory, wherein the computer-executable instructions, when executed by the processor, configure the processor to:
receive a histological image including a first two-dimensional array of pixels;
generate, using a first convolutional neural network, a first output image mapped to the histological image and one of a plurality of relevance classes assigned to each pixel of the first output image, the first convolutional neural network being trained based on a training data set including (a) histological images and (b) pathologist interaction data, wherein the plurality of relevance classes includes at least one class representing a pixel of interest and at least one class representing a pixel that is not of interest, and wherein the pathologist interaction data comprise parameters relating to how pathologists have interacted with a plurality of other histological images; and
generate from the first output image a segmentation mask in which areas of interest occupied by pixels of interest are marked;
apply a second convolutional neural network to the histological image to generate a second output image with a second two-dimensional array of pixels with a mapping to that of the histology image, wherein the histological image is of a tissue sample that has been treated with a test compound, and wherein the second convolutional network has been trained with a training data set comprising a plurality of histological images of tissue samples that have not been treated with the test compound;
compute a distance between the second output image and the corresponding portion of the histological image in accordance with the mapping;
generate a toxicity map for the histological image based on the computed distances;
analyze the segmentation map and the toxicity map to identify areas of the histological image that are marked as areas of interest in the segmentation map and that are identified as exhibiting toxicity in the toxicity map; and
increase a toxicity confidence score for each area of the histological image that is marked as of interest in the segmentation map and is also identified as exhibiting toxicity in the toxicity map.

2. The apparatus of claim 1 , further comprising:
a visualization application operable to create visualizations of histological images having regard to their toxicity maps; and
a display configured to receive visualizations from the visualization application.

3. The apparatus of claim 2 , wherein the visualization application provides a user interface toxic area selection control operable to permit a user to interact with a visualization so as to select a toxic area.

4. The apparatus of claim 3 , wherein the toxic area selection control has a scroll function for sweeping through the toxic areas in order of ranking.

5. The apparatus of claim 3 , further comprising: a data repository configured to store records of patient data including histological images with associated toxicity maps; and network connections enabling transfer of patient data records or parts thereof between the computer apparatus and the data repository.

6. The apparatus of claim 1 , wherein the computer-executable instructions, when executed, further configure the processor to identify one or more areas which are marked as of interest in the segmentation map and are also identified as exhibiting toxicity in the toxicity map as toxic.

7. The apparatus of claim 6 , wherein the toxicity map includes a heat map in which areas identified as toxic are assigned a temperature value proportional to their distance value.

8. The apparatus of claim 7 , wherein the computer-executable instructions, when executed, further configure the processor to apply a segmentation algorithm to the toxicity map to group areas identified as toxic and thereby generate a segmentation mask of toxic areas.

9. A method comprising:
receiving a histological image including a first two-dimensional array of pixels;
generating, using a first convolutional neural network, a first output image mapped to the histological image and one of a plurality of relevance classes assigned to each pixel of the first output image, the first convolutional neural network being trained based on a training data set including (a) histological images and (b) pathologist interaction data, wherein the plurality of relevance classes includes at least one class representing a pixel of interest and at least one class representing a pixel that is not of interest, and wherein the pathologist interaction data comprise parameters relating to how pathologists have interacted with a plurality of other histological images; and
generating from the first output image a segmentation mask in which areas of interest occupied by pixels of interest are marked;
applying a second convolutional neural network to the histological image to generate a second output image with a second two-dimensional array of pixels with a mapping to that of the histology image, wherein the histological image is of a tissue sample that has been treated with a test compound, and wherein the second convolutional network has been trained with a training data set comprising a plurality of histological images of tissue samples that have not been treated with the test compound;
computing a distance between the second output image and the corresponding portion of the histological image in accordance with the mapping;
generating a toxicity map for the histological image based on the computed distances;
analyzing the segmentation map and the toxicity map to identify areas of the histological image that are marked as areas of interest in the segmentation map and that are identified as exhibiting toxicity in the toxicity map; and
increasing a toxicity confidence score for each area of the histological image that is marked as of interest in the segmentation map and is also identified as exhibiting toxicity in the toxicity map.

10. The method of claim 9 , wherein the method comprises identifying one or more areas which are marked as of interest in the segmentation map and are also identified as exhibiting toxicity in the toxicity map as toxic.

11. The method of claim 10 , wherein the toxicity map includes a heat map in which areas identified as toxic are assigned a temperature value proportional to their distance value.

12. The method of claim 10 , further comprising applying a segmentation algorithm to the toxicity map to group areas identified as toxic and thereby generate a second segmentation mask, wherein the second segmentation mask depicts toxic areas.

13. The method of claim 10 , further comprising saving an overall toxicity label, wherein the overall toxicity label is a binary label designating the histological image as toxic if any area in the histological image has been identified as toxic and non-toxic if no area in the histological image has been identified as toxic.

14. The method of claim 9 , further comprising:
providing a visualization application; and
creating a visualization of the histological image having regard to the toxicity map.

15. The method of claim 14 , wherein the visualization includes an overview viewing pane in which the toxicity map is overlaid on the histological image.

16. The method of claim 14 , wherein the visualization includes respective overview viewing panes in which the toxicity map and the histological image are presented adjacent each other for one-to-one comparison.

17. The method of

Assignees

• Leica Biosystems Imaging Inc

Inventors

• Georgescu Walter
• Saligrama Kiran
• Olson Allen
• Mallya Udupi Girish
• Oliveira Bruno

Classifications

• G06V10/82

  using neural networks · CPC title

• G06V10/764

  using classification, e.g. of video objects · CPC title

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title

• G06T2207/10056

  Microscopic image · CPC title

• G06T7/0012 Primary

  Biomedical image inspection · CPC title