End-to-end attention pooling-based classification method for histopathology images

Patent metadata

Field	Value
Publication number	US-12008076-B2
Application number	US-202117546965-A
Country	US
Kind code	B2
Filing date	Dec 9, 2021
Priority date	Dec 10, 2020
Publication date	Jun 11, 2024
Grant date	Jun 11, 2024

Abstract

Provided is an end-to-end attention pooling-based classification method for histopathological images obtaining a better classification effect for small number of samples by S1, cutting the histopathology image into patches of a specified size, removing the patches with too much background area and packaging the remaining patches into a bag; S2, training a deep learning network by taking the bag obtained in S1 as an input using a standard multi-instance learning method; S3, scoring all the patches by using the trained deep learning network, and selecting m patches with highest and lowest scores for each whole slide image to form a new bag; S4, building a deep learning network including an attention pooling module, and training the network by using the new bag obtained in S3; and S5, after the histopathology image to be classified is processed in S1 and S3, performing classification by using the model obtained in S4.

First claim

What is claimed is:

1. An end-to-end attention pooling-based classification method for histopathology images, comprising the following steps:
S1, cutting a histopathological whole slide image into square patches with a side length L, pre-processing to filter patches, and packaging the remaining patches into a bag;
S2, modifying the last fully connected layer in a pre-trained Resnet50 network, recording the modified model as A, and training the model A by a standard multi-instance learning method;
S3, scoring all the patches by using the trained model A, and selecting m patches with highest and lowest scores for each image to form a new bag;
S4, building a deep learning model B comprising an attention pooling module, and training the model B by using the new bag obtained in S3; and
S5, after a bag containing 2 m patches is obtained by processing the histopathology image to be classified in S1 and S3, classifying the bag by using the model B trained in S4, wherein the classification result is the final classification result of the image to be classified.

2. The end-to-end attention pooling-based classification method for histopathology images according to claim 1 , wherein preprocessing to filter patches in S1 refers to removing the patches with the background area ratio exceeding a certain range, and is specifically implemented as follows:
firstly, calculating a threshold of the background area and foreground area for a whole slide image at a low resolution by using an Otsu method, wherein a specific algorithm of the Otsu method is to find a threshold t to minimize a sum of variances within the foreground area and the background area, and a calculation formula is as follows:
σ 2 ( t )= w 0 ( t )σ 0 2 ( t )+ w 1 ( t )σ 1 2 ( t ),

σ 0 2 (t) refers to a variance of a gray value within the background area when t is taken as the threshold, and σ 1 2 (t) refers to a variance within the foreground area when t is taken as the threshold; and it is assumed that when t is taken as the threshold, a gray value of pixels belonging to the background area is recorded as B(i), a gray value of pixels belonging to the foreground area is recorded as F(i), a number of the pixels belonging to the foreground area is recorded as N F , and a number of the pixels belonging to the background area is recorded as N B , and calculation methods of the variances are as follows:

σ
0
2

⁡

(
t
)

=

1

N
B

⁢

∑

(

B
⁡

(
i
)

-

1

N
B

⁢

∑

B
⁡

(
i
)

)

2

,
and

σ
1
2

⁡

(
t
)

=

1

N
F

⁢

∑

(

F
⁡

(
i
)

-

1

N
F

⁢

∑

F
⁡

(
i
)

)

2

,

and
w 1 (t) and w 2 (t) are proportions of the foreground area and the background area when t is taken as the threshold; and after an optimal threshold is obtained by using the above algorithm, a proportion of the background area in a whole patch area under the optimal threshold is calculated, and if the proportion exceeds a certain value, the patch is discarded.

3. The end-to-end attention pooling-based classification method for histopathology images according to claim 1 , wherein in S2, an output dimension of the last fully connected layer of the Resnet50 network is modified to 128, and a fully connected layer with an input dimension of 128 and an output dimension of 2 is added after the fully connected layer of the last layer.

4. The end-to-end attention pooling-based classification method for histopathology images according to claim 1 , wherein training the model A by the standard multi-instance learning method in S2 is as follows: an assumption of multi-instance learning is that at least one patch in a positive bag is positive, and all patches in a negative bag are negative; before each epoch of training, firstly all the patches are scored by using the model A, k patches with the highest score in each image are selected, and the selected patches are labeled the same as the whole slide image; and all the selected patch-label pairs constitute a dataset required for training, the model A is trained, and the above process is repeated until an accuracy of the model A on a validatio

Assignees

• Univ Wuhan

Inventors

• Liu Juan
• Zuo Zhiqun
• Chen Yuqi
• Li Zhuoyu
• Feng Jing

Classifications

• G06N3/09

  Supervised learning · CPC title

• G06N3/0895

  Weakly supervised learning, e.g. semi-supervised or self-supervised learning · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06F18/241

  relating to the classification model, e.g. parametric or non-parametric approaches · CPC title

• G06T2207/30068

  Mammography; Breast · CPC title