Systems and methods for detection of structures and/or patterns in images

Patent metadata

Field	Value
Publication number	US-10109052-B2
Application number	US-201615360447-A
Country	US
Kind code	B2
Filing date	Nov 23, 2016
Priority date	May 23, 2014
Publication date	Oct 23, 2018
Grant date	Oct 23, 2018

Abstract

The subject disclosure presents systems and computer-implemented methods for automatic immune cell detection that is of assistance in clinical immune profile studies. The automatic immune cell detection method involves retrieving a plurality of image channels from a multi-channel image such as an RGB image or biologically meaningful unmixed image. A cell detector is trained to identify the immune cells by a convolutional neural network in one or multiple image channels. Further, the automatic immune cell detection algorithm involves utilizing a non-maximum suppression algorithm to obtain the immune cell coordinates from a probability map of immune cell presence possibility generated from the convolutional neural network classifier.

First claim

What is claimed is:

1. An image processing method for automatic detection of biological structures in a multi-channel image obtained from a biological tissue sample being stained by multiple stains, the method comprising:
unmixing the multi-channel image to provide an unmixed image per channel, each channel being representative of one of the biological structures,
detecting of candidate locations for the biological structures in the unmixed images by applying an image processing algorithm,
for each of the candidate locations, extracting a stack of image patches having a predefined size from the unmixed images, the image patches of the stack comprising the candidate location, each one of the stacks of image patches comprising one image patch per channel,
sequentially entering the stacks of image patches into a trained convolutional neural network, the convolutional neural network comprising at least convolutional layers and sub-sampling layers in alternating order, the first one of the convolutional layers being coupled to inputs of the convolutional neural network, each one of the inputs being assigned to one of the channels, the first one of the convolutional layers having a number of feature maps, the convolutional neural network being configured for connection mapping of the inputs to the feature maps of the first one of the convolutional layers using co-location data being descriptive of groups of the stains, each group comprising co-located stains, in order to map sub-sets of the channels that are representative of co-located biological features to a common feature map,
outputting a probability map representing a probability for the presence of the biological features in the multi-channel image from an output of the convolutional neural network.

2. The method of claim 1 , the number of feature maps being below the number of channels.

3. The method of claim 1 , the convolutional neural network having its final convolutional layer coupled to a full connection layer that outputs the probability map.

4. The method of claim 1 , further comprising staining the biological tissue sample with multiple stains to provide the channels, and
acquiring the multi-channel image using an image sensor.

5. The method of claim 4 , the image sensor having a number of color channels below the number of channels of the multi-channel image, wherein the unmixing of the multi-channel image is performed using the co-location data.

6. The method of claim 1 , the convolutional neural network being trained by:
acquiring a multi-channel training image from a training biological tissue sample being stained by the multiple stains,
unmixing the multi-channel training image to provide an unmixed image per channel,
displaying the unmixed training images on a user interface,
receiving labeling information indicative of the presence and locations of the biological structures in the multi-channel training image,
for each of the locations indicated by the labeling information, extracting a stack of training image patches having the predefined size from the unmixed training images, the image patches comprising the indicated location,
training the convolutional neural network by sequentially inputting the stacks of training image patches, wherein the probability map that is outputted by the convolutional neural network in response to inputting the training image patches is compared to the labeling information for training of the convolutional neural network.

7. An image processing system for automatic detection of biological structures in a multi-channel image obtained from a biological tissue sample being stained by multiple stains comprising:
an unmixing component configured to unmix the multi-channel image to provide an unmixed image per channel, each channel being representative of one of the biological structures,
a detection component configured to detect candidate locations for the biological structures in the unmixed images by applying an image processing algorithm,
a patch extraction component configured to process each of the candidate locations by extracting stack of image patches having a predefined size from the unmixed images, the image patches of the same stack comprising the respective candidate location, each one of the stacks of image patches comprising one image patch per channel,
a trained convolutional neural network for sequential entry of the stacks of image patches, the convolutional neural network comprising at least convolutional layers and sub-sampling layers in alternating order, the first one of the convolutional layers being coupled to inputs of the convolutional neural network, each one of the inputs being assigned to one of the channels, the first one of the convolutional layers being configured to generate a number of feature maps from the stacks of image patches, the convolutional neural network being configured for connection mapping of the inputs to feature maps of the first one of the convolutional layers using co-location data being descriptive of groups of the stains, each group comprising co-located stains, in order to map sub-sets of the channels that are representative of co-located biological features to a common feature map,
an output configured to output a probability map representing a probability for the presence of the biological features in the multi-channel image from an output of the convolutional neural network.

8. The image processing system of claim 7 , the number of feature maps being below the number of channels.

9. The image processing system of claim 7 , the convolutional neural network having its final convolutional layer coupled to a full connection layer that outputs the probability map.

10. The image processing system of claim 7 , further comprising:
a staining component for staining the biological tissue sample with multiple stains to provide the channels, and
an acquisition component for acquiring the multi-channel image using an image sensor that has a number of color channels below the number of channels of the multi-channel image, the unmixing component being configured to perform the unmixing of the multi-channel image using the co-location data.

Assignees

• Ventana Med Syst Inc

Inventors

• Chefd'Hotel Christophe
• Chen Ting

Classifications

• G06N3/084

  Backpropagation, e.g. using gradient descent · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title

• G06T2207/10024

  Color image · CPC title

• G06T7/0012 Primary

  Biomedical image inspection · CPC title