Image pre-processing for accelerating cytological image classification by fully convolutional neural networks

Patent metadata

Field	Value
Publication number	US-10586336-B2
Application number	US-201815983434-A
Country	US
Kind code	B2
Filing date	May 18, 2018
Priority date	May 18, 2018
Publication date	Mar 10, 2020
Grant date	Mar 10, 2020

Abstract

A fully convolutional network (FCN) implemented on a specialized processor optimized for convolution computation can achieve a speed-up in cell classification. Without re-optimizing the specialized processor, a further speed-up is achieved by compacting a testing image of cells, and processing the compacted testing image with the FCN. The testing image is first segmented into a background and regions of interest (ROIs). The ROIs are packed closer together by rearranging the ROIs without resizing them under a constraint that any two adjacent rearranged ROIs are separated by a distance in pixel not less than a minimum distance determined according to stride values of FCN convolutional layers. Geometrical operations in ROI rearrangement include relocating the ROIs and, optionally, rotating the ROIs. The rearranged ROIs are enclosed by a boundary, typically a rectangular boundary, to form the compacted testing image having an area smaller than that of the testing image.

First claim

What is claimed is:

1. A method for classifying a plurality of cells imaged on a testing image by using a fully convolutional network (FCN), the FCN having plural convolutional layers each having a respective value of stride, the FCN being implemented on a specialized processor having a hardware configuration optimized for computing plural convolutional products in parallel for an image, the method comprising:
segmenting the testing image into a background and plural regions of interest (ROIs), an individual ROI comprising one or more connected individual cells disjoint from remaining cells in the plurality of cells;
compacting the testing image to form a compacted testing image, including:
rearranging the ROIs for packing the ROIs closer together under a first constraint that any adjacent two of the rearranged ROIs are separated in each of x- and y-directions of the testing image by a distance in pixel not less than a minimum distance determined according to the stride values of the convolutional layers, wherein an individual ROI is rearranged by performing one or more geometric operations without resizing the individual ROI, the one or more geometric operations including relocating the individual ROI; and
enclosing an entirety of rearranged ROIs with a boundary to form the compacted testing image, wherein the boundary is a perimeter of the compacted testing image and is selected under a second constraint that a first number of pixels occupied by the compacted testing image is less than a second number of pixels occupied by the testing image; and

classifying the plurality of cells by processing the compacted testing image rather than the original testing image with the FCN for reducing a time required to accomplish the classifying of the plurality of cells without a need for re-optimizing the hardware configuration.

2. The method of claim 1 , wherein the minimum distance, d min , is given by d min =½Π i=1 N φ i where N is a total number of the convolutional layers, and φ i is the stride value of ith convolutional layer.

3. The method of claim 1 further comprising:
before the testing image is compacted, replacing the background with a blank one in the testing image for minimizing interference due to the background in the classifying of the plurality of cells.

4. The method of claim 1 , wherein the boundary is a rectangular boundary.

5. The method of claim 4 , wherein the rectangular boundary has a width and a length measured in pixel, each of the width and length being selected to be minimally sufficient to enclose all the rearranged ROIs.

6. The method of claim 4 , wherein the rectangular boundary has a width and a length measured in pixel, each of the width and length being selected to be minimally sufficient to enclose all the rearranged ROIs while satisfying one or more image-size requirements of the FCN in processing the compacted testing image.

7. The method of claim 1 , wherein:
the segmenting of the testing image into the background and the ROIs includes determining a location and a contour of each of the ROIs on the testing image; and
the rearranging of the ROIs includes:
gridding the testing image with a grid unit to form a gridded image, wherein the individual ROI is mosaicked to form a corresponding ROI grid on the gridded image, whereby plural ROI grids are formed on the gridded image;
relocating the ROI grids on the gridded image one by one according to a descending order of ROI-grid size under a third constraint that the ROI grids after relocation do not overlap; and
in relocating the corresponding ROI grid on the gridded image with a directed displacement, relocating the individual ROI on the testing image with the same directed displacement.

8. The method of claim 7 , wherein the grid unit is a minimum grid unit.

9. The method of claim 7 , wherein the relocating of the ROI grids on the gridded image one by one according to the descending order of ROI-grid size under the third constraint comprises:
creating a second gridded image having a dimension of the gridded image and being empty when created;
copying the ROI grids one by one to the second gridded image according to the descending order of ROI-grid size, wherein the copying of said corresponding ROI grid to the second gridded image includes:
dilating said corresponding ROI grid by one grid unit to form a dilated ROI grid;
bounding the dilated ROI grid with a minimum bounding rectangle to form a rectangular window, wherein the rectangular window contains a first copy of said corresponding ROI grid;
sliding the rectangular window on the second gridded image in a raster scanning manner along an x- or y-direction to identify a fitted region on the second gridded image such that in the fitted region, said first copy does not overlap with another ROI-grid copy already on the second gridded image; and
putting said first copy on the fitted region, whereby a first location of said first copy on the second gridded image is same as a second location on the gridded image for said corresponding ROI grid to be relocated, thereby allowing the directed displacement to be determined; and

relocating said corresponding ROI grid to the second location on the gridded image.

10. The method of claim 9 , wherein the fitted region has a size of the rectangular window, and the fitted region is identified by an identifying process that includes determining whether a candidate region on the second gridded image is the fitted region, the candidate region having the size of the rectangular window, the determining of whether the candidate region is the fitted region comprising:
on the rectangular window, assigning each of squares occupied by the dilated ROI grid with a value of 1, and each of remaining squares a value of 0, whereby a first array of values assigned to the rectangular window is obtained;
on the candidate region, assigning each of squares occupied by any ROI-grid copy already on the second gridded image with a value of 1, and each of remaining squares a value of 0, whereby a second array of values assigned to the candidate region is obtained;
performing an element-wise Boolean OR operation between the first array of values and the second array of values to yield a third array of values;
computing a first sum by summing the values in the first array;
computing a second sum by summing the values in the second array;
computing a third sum by summing the values in the third array;
computing a fourth sum by adding the first sum and the second sum together;
responsive to finding that the third sum is equal to the fourth sum, assigning the candidate region as the fitted region, whereby the fitted region is identified; and
responsive to finding that the third sum is not equal to the fourth sum, declaring that the candidate region is not the fitted region.

11. The method of claim 1 , wherein the one or more geometric operations further include rotating the individual ROI before relocation.

12. The method of claim 1 , wherein:
the testing image is a color one having plural color channels, each of the color channels having respective luminance data; and
the segmenting of the testing image into the background and the ROIs includes:
determining a threshold of luminance value for differentiating the ROIs from the background on the testing image;
performing thresholding on each of the color channels according to the determined threshold to yield a respective binary image, whereby plural binary images for the color channels are obtained;
performing a pixel-wise Boolean operation on the binary images to yield a mask, each pixel of the mask taking either a first value or a second value, the mask co

Assignees

• Hong Kong Applied Science & Tech Research Inst Co Ltd

Inventors

• Hu Yu
• Wang Lu
• Leung Ping Shun

Classifications

• G06N3/08

  Learning methods · CPC title

• G06V20/695

  Preprocessing, e.g. image segmentation · CPC title

• G16H50/20 Primary

  for computer-aided diagnosis, e.g. based on medical expert systems · CPC title

• G06T7/11 Primary

  Region-based segmentation · CPC title

• G06T7/194

  involving foreground-background segmentation · CPC title