Automatic detection of mitosis using handcrafted and convolutional neural network features

Patent metadata

Field	Value
Publication number	US-9430829-B2
Application number	US-201414562883-A
Country	US
Kind code	B2
Filing date	Dec 8, 2014
Priority date	Jan 30, 2014
Publication date	Aug 30, 2016
Grant date	Aug 30, 2016

Abstract

One example apparatus associated with detecting mitosis in breast cancer pathology images by combining handcrafted (HC) and convolutional neural network (CNN) features in a cascaded architecture includes a set of logics that acquires an image of a region of tissue, partitions the image into candidate patches, generates a first probability that the patch is mitotic using an HC feature set and a second probability that the patch is mitotic using a CNN-learned feature set, and classifies the patch based on the first probability and the second probability. If the first and second probabilities do not agree, the apparatus trains a cascaded classifier on the CNN-learned feature set and the HC feature set, generates a third probability that the patch is mitotic, and classifies the patch based on a weighted average of the first probability, the second probability, and the third probability.

First claim

What is claimed is:

1. A non-transitory computer-readable storage medium storing computer-executable instructions that when executed by a computer control the computer to perform a method for detecting cellular mitosis in a region of cancerous tissue, the method comprising:
acquiring an image of cancerous tissue;
segmenting the image into a candidate mitosis patch;
extracting a set of convolutional neural network (CNN) learned features from the candidate mitosis patch using a CNN;
training a CNN classifier using the set of CNN-learned features;
generating a CNN classification score by classifying the candidate mitosis patch with the CNN classifier;
extracting a set of hand-crafted (HC) features from the candidate mitosis patch;
training an HC classifier using the set of HC features;
generating an HC classification score by classifying the candidate mitosis patch with the HC classifier;
producing a final classification based, at least in part, on both the CNN classification score and the HC classification score;
controlling an automated mitotic nuclei detection system to classify the candidate mitosis patch as mitotic or non-mitotic based on the final classification;
generating a mitotic count by summing the number of candidate mitosis patches classified as mitotic by the automated mitotic nuclei detection system; and
controlling an automated cancer grading system to grade the image using a Bloom-Richardson grade, where the Bloom-Richardson grade is based, at least in part, on the mitotic count, where producing the final classification comprises:
comparing the CNN classification score to the HC classification score, and
upon determining that the CNN classification score and the HC classification score are not within a threshold range:
training a cascaded classifier using a stacked set of features, where the stacked set of features comprises the set of CNN-learned features and the set of HC features;
generating a cascaded classification score by classifying the candidate mitosis patch with the cascaded classifier, and

producing a final classification, based, at least in part, on a weighted average of the CNN classification score, the HC classification score, and the cascaded classification score, where the final classification indicates the probability that the mitosis patch is mitotic.

2. The non-transitory computer-readable storage medium of claim 1 , where producing the final classification comprises:
comparing the CNN classification score to the HC classification score, and
upon determining that the CNN classification score and the HC classification score are within a threshold range:
producing a final classification, based, at least in part, on the CNN classification score and the HC classification score, where the final classification indicates the probability that the mitosis patch is mitotic.

3. The non-transitory computer-readable storage medium of claim 1 , where acquiring the image comprises scanning an image from a high power field (HPF) of a hematoxylin and eosin (H&E) stained tissue slide, where the HPF represents at least a 512×512 μm region of tissue, where the HPF is acquired using a slide scanner and a multi-spectral microscope, where the image is a RGB color space image, and where the image has dimensions of at least 2084 pixels×2084 pixels.

4. The non-transitory computer-readable storage medium of claim 3 , where segmenting the image into a candidate mitosis patch comprises:
generating a blue-ratio image by converting the image from RBG color space to a blue-ratio color space;
computing a Laplacian of Gaussian (LoG) response for the blue-ratio image, and
identifying a candidate nuclei by integrating globally fixed thresholding and local dynamic thresholding.

5. The non-transitory computer-readable storage medium of claim 4 , where the CNN comprises:
a first convolutional layer that has at least P units, P being a number;
a first pooling layer connected to the first convolutional layer, where the first pooling layer has at least Q units, Q being a number;
a second convolutional layer connected to the first pooling layer, where the second convolutional layer has at least X units, X being a number greater than P and greater than Q;
a second pooling layer connected to the second convolutional layer, where the second pooling layer has Y units, Y being a number greater than P and greater than Q;
a fully-connected layer connected to the second pooling layer, where the fully-connected layer has Z units, Z being a number greater than Y and greater than X, and
an output layer connected to the fully-connected layer, where the output layer has at least two output units.

6. The non-transitory computer-readable storage medium of claim 5 , where P is at least 64, where Q is at least 64, where X is at least 128, where Y is at least 128, and where Z is at least 256.

7. The non-transitory computer-readable storage medium of claim 6 , where extracting the set of CNN learned features from the candidate mitosis patch comprises:
generating a YUV color space image by converting the image from RGB color space to YUV color space and by normalizing the YUV color space image to a mean of zero and a variance of one;
extracting an input feature map from the YUV color space image;
generating a first output feature map by applying, in the first convolution layer of the CNN, a two dimensional (2D) convolution of the input feature map and a first convolution kernel;
generating a first pooled map by applying, in the first pooling layer in the CNN, an L 2 pooling function over a spatial window applied over the first output feature map, where the L 2 pooling function is applied without overlapping;
generating a second output feature map by applying, in the second convolution layer of the CNN, a 2D convolution of the first pooled map and a second convolution kernel;
generating a second pooled map by applying, in the second pooling layer of the CNN, an L 2 pooling function over a spatial window applied over the second output feature map, where the L 2 pooling function is applied without overlapping;
generating a feature vector by applying the second pooled map to the fully-connected layer of the CNN, and
generating a fully-connected layer output by activating an output unit in the CNN based, at least in part, on a logistic regression model and the feature vector, where the output unit is one of a mitosis unit or a non-mitosis unit.

8. The non-transitory computer-readable storage medium of claim 7 , where training the CNN classifier using the set of CNN-learned features comprises:
generating a rotated mitosis patch by rotating the candidate mitosis patch;
generating a mirrored mitosis patch by mirroring the candidate mitosis patch, and
computing a log-likelihood that the candidate mitosis patch is mitotic, where computing the log-likelihood comprises minimizing a loss function by training the CNN classifier using the rotated mitosis patch, the mirrored mitosis patch, and the candidate mitosis patch using a Stochastic Gradient Descent, where the loss function is described by:

L
⁡

(
x
)

=

-

log
⁡

[

e

x
i

Assignees

• Univ Case Western Reserve

Inventors

• Madabhushi Anant
• Wang Haibo
• Cruz-Roa Angel
• Gonzalez Fabio

Classifications

• G06V10/764

  using classification, e.g. of video objects · CPC title

• G06F18/24317

  Piecewise classification, i.e. whereby each classification requires several discriminant rules · CPC title

• G06T7/0012 Primary

  Biomedical image inspection · CPC title

• G06T2207/10056

  Microscopic image · CPC title

• G06K9/6281

  Physics · mapped topic