Computer supported review of tumors in histology images and post operative tumor margin assessment

Patent metadata

Field	Value
Publication number	US-12243231-B2
Application number	US-202318543461-A
Country	US
Kind code	B2
Filing date	Dec 18, 2023
Priority date	May 29, 2019
Publication date	Mar 4, 2025
Grant date	Mar 4, 2025

Abstract

A computer apparatus and method for identifying and visualizing tumors in a histological image and measuring a tumor margin are provided. A CNN is used to classify pixels in the image according to whether they are determined to relate to nontumorous tissue, or one or more classes for tumorous tissue. Segmentation is carried out based on the CNN results to generate a mask that marks areas occupied by individual tumors. Summary statistics for each tumor are computed and supplied to a filter which edits the segmentation mask by filtering out tumors deemed to be insignificant. Optionally, the tumors that pass the filter may be ranked according to the summary statistics, for example in order of clinical relevance or by a sensible order of review for a pathologist. A visualization application can then display the histological image having regard to the segmentation mask, summary statistics and/or ranking. Tumor masses extracted by resection are painted with an ink to highlight its surface region. The CNN is trained to distinguish ink and no-ink tissue as well as tumor and no-tumor tissue. The CNN is applied to the histological image to generate an output image whose pixels are assigned to the tissue classes. Tumor margin status of the tissue section is determined by the presence or absence of tumor-and-ink classified pixels. Tumor margin involvement and tumor margin distance are determined by computing additional parameters based on classification-specified inter-pixel distance parameters.

First claim

What is claimed is:

1. A computer apparatus for identifying tumors in a histological image, comprising:
a memory configured to store computer-executable instructions; and
a hardware processor in communication with the memory, wherein the computer-executable instructions, when executed by the processor, configure the processor to perform a method comprising:
generating, from an output image, a segmentation mask having areas occupied by individual tumors in the output image marked in the segmentation mask;
computing statistics for each tumor marked in the segmentation mask; and
applying a filter to the statistics of the tumors to edit the segmentation mask, wherein application of the filter selects and deselects tumors according to the filter to edit the segmentation mask to remove insignificant tumors; wherein:
the output image comprises an array of pixels, each pixel in the output image being assigned to one of a plurality of tissue classes;
the output image is generated by a convolutional neural network based on the histological image by performing acts comprising:
extracting image patches from the histological image, the image patches being area portions of the histological image having a size defined by numbers of pixels in width and height;
providing the convolutional neural network with a set of weights and a plurality of channels, each channel corresponding to one of the plurality of tissue classes;
inputting each image patch as an input image patch into the convolutional neural network;
performing multi-stage convolution to generate convolution layers of ever decreasing dimensions up to and including a final convolution layer of minimum dimensions, followed by multi-stage transpose convolution to reverse the convolutions by generating deconvolution layers of ever increasing dimensions until a layer is recovered matched in size to the input image patch, each pixel in the recovered layer containing a probability of belonging to each of the tissue classes; and
assigning the tissue class to each pixel of the recovered layer based on said probabilities to arrive at an output image patch.

2. The apparatus of claim 1 , wherein the method further comprises:
providing the convolutional neural network with at least one skip connection, each of which takes intermediate results from at least one of the convolution layers of larger dimensions than the final convolution layer and subjects those results to as many transpose convolutions as needed, which may be none, one or more than one, to obtain at least one further recovered layer matched in size to the input image patch; and
prior to said step of assigning a tissue class to each pixel, further processing the recovered layer to combine it with the at least one further recovered layers in order to recompute the probabilities to take account of the at least one skip connection.

3. A computer apparatus for identifying tumors in a histological image, comprising:
a memory configured to store computer-executable instructions; and
a hardware processor in communication with the memory, wherein the computer-executable instructions, when executed by the processor, configure the processor to perform a method comprising:
generating, from an output image, a segmentation mask having areas occupied by individual tumors in the output image marked in the segmentation mask;
computing statistics for each tumor marked in the segmentation mask; and
applying a filter to the statistics of the tumors to edit the segmentation mask, wherein application of the filter selects and deselects tumors according to the filter to edit the segmentation mask to remove insignificant tumors;

wherein the histological image is a composite including a plurality of histological images obtained from differently stained, adjacent sections of a region of tissue.

4. A computer apparatus for identifying tumors in a histological image, comprising:
a memory configured to store computer-executable instructions; and
a hardware processor in communication with the memory, wherein the computer-executable instructions, when executed by the processor, configure the processor to perform a method comprising:
generating, from an output image, a segmentation mask having areas occupied by individual tumors in the output image marked in the segmentation mask;
computing statistics for each tumor marked in the segmentation mask; and
applying a filter to the statistics of the tumors to edit the segmentation mask, wherein application of the filter selects and deselects tumors according to the filter to edit the segmentation mask to remove insignificant tumors;

wherein the method further comprises:
creating a visualization of the histological image in accordance with the edited segmentation mask;
receiving a selection of an individual tumor in the edited segmentation mask; and
executing an additional computational diagnostic process on the selected individual tumor.

5. A method for identifying tumors in a histological image, comprising:
generating, from an output image, a segmentation mask having areas occupied by individual tumors in the output image marked in the segmentation mask;
computing statistics for each tumor marked in the segmentation mask; and
applying a filter to the statistics of the tumors to edit the segmentation mask, wherein application of the filter selects and deselects tumors according to the filter to edit the segmentation mask to remove insignificant tumors;
wherein:
the output image comprises an array of pixels, each pixel in the output image being assigned to one of a plurality of tissue classes;
the output image is generated by a convolutional neural network based on the histological image by performing acts comprising:
extracting image patches from the histological image, the image patches being area portions of the histological image having a size defined by numbers of pixels in width and height;
providing the convolutional neural network with a set of weights and a plurality of channels, each channel corresponding to one of the plurality of tissue classes;
inputting each image patch as an input image patch into the convolutional neural network;
performing multi-stage convolution to generate convolution layers of ever decreasing dimensions up to and including a final convolution layer of minimum dimensions, followed by multi-stage transpose convolution to reverse the convolutions by generating deconvolution layers of ever increasing dimensions until a layer is recovered matched in size to the input image patch, each pixel in the recovered layer containing a probability of belonging to each of the tissue classes; and
assigning the tissue class to each pixel of the recovered layer based on said probabilities to arrive at an output image patch.

6. The method of claim 5 , wherein the method further comprises:
providing the convolutional neural network with at least one skip connection, each of which takes intermediate results from at least one of the convolution layers of larger dimensions than the final convolution layer and subjects those results to as many transpose convolutions as needed, which may be none, one or more than one, to obtain at least one further recovered layer matched in size to the input image patch; and
prior to said step of assigning a tissue class to each pixel, further processing the recovered layer to combine it with the at least one further recovered layers in order to recompute the probabilities to take account of the at least one skip connection.

7. A method for identifying tumors in a histological image, comprising:
generating, from an output image, a segmentation mask having areas occupied by individual tumors in the output image marked in the segmentation mask;
computing statistics fo

Assignees

• Leica Biosystems Imaging Inc

Inventors

• Georgescu Walter

Classifications

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06F18/2148

  characterised by the process organisation or structure, e.g. boosting cascade · CPC title

• G06F18/24

  Classification techniques · CPC title

• G06T2207/30096

  Tumor; Lesion · CPC title