Weakly supervised learning with whole slide images

Patent metadata

Field	Value
Publication number	US-11954596-B2
Application number	US-202017605224-A
Country	US
Kind code	B2
Filing date	Mar 10, 2020
Priority date	Apr 25, 2019
Publication date	Apr 9, 2024
Grant date	Apr 9, 2024

Abstract

Techniques are provided for determining classifications based on WSIs. A varied-size feature map is generated for each training WSI by generating a grid of patches for the training WSI, segmenting the training WSI into tissue and non-tissue areas, and converting patches comprising the tissue areas into tensors. Bounding boxes are generated based on the patches comprising tissue areas and segmented into feature map patches. A fixed-size feature map is generated based on a subset of the feature map patches. A classifier model is trained to process fixed-size feature maps corresponding to the training WSIs such that, for each fixed-size feature map, the classifier model is operable to assign a WSI-level tissue or cell morphology classification or regression based on the tensors. A classification engine is configured to use the trained classifier model to determine a WSI-level tissue or cell morphology classification or regression for a test WSI.

First claim

We claim:

1. A computerized method of determining classifications based on whole slide images (WSIs), comprising:
obtaining a plurality of training WSIs;
generating a varied-size feature map for each of the plurality of training WSIs by generating a grid of patches for the training WSI, segmenting the training WSI into tissue and non-tissue areas, and converting patches comprising the tissue areas into tensors;
generating at least one bounding box based on the patches comprising the tissue areas;
segmenting the at least one bounding box into feature map patches;
generating a fixed-size feature map based on at least a subset of the feature map patches;
configuring a classifier model to process fixed-size feature maps corresponding to the training WSIs such that, for each fixed-size feature map, the classifier model is operable to assign a WSI-level tissue or cell morphology classification or regression based on the tensors;
training the classifier model using the fixed-size feature map corresponding to the plurality of training WSIs; and
configuring a classification engine to use the trained classifier model to determine a WSI-level tissue or cell morphology classification or regression for a test WSI.

2. The method of claim 1 , wherein the plurality of training WSIs comprises less than 1000 pathology slide images.

3. The method of claim 1 , wherein the plurality of training WSIs comprises hematoxylin and eosin (H&E)-stained whole slide images.

4. The method of claim 1 , wherein each of the plurality of training WSIs corresponds to an WSI-level label indicating a classification comprising at least one of a type of cancer and a cancer grade.

5. The method of claim 1 , wherein each of the plurality of training WSIs corresponds to an WSI-level label indicating a regression comprising at least one of a percentage of tumor-infiltrating lymphocytes, RNA expression, mutation burden, and allele frequency.

6. The method of claim 1 , further comprising converting patches of the grid of patches determined to comprise non-tissue areas into tensors comprising white feature components.

7. The method of claim 1 , further comprising:
filtering the grid of patches for a minimum color variance; and
eliminating each patch determined to be empty space or background from further processing based on the filtering.

8. The method of claim 1 , wherein each of the tensors comprises a multidimensional descriptive vector.

9. The method of claim 8 , wherein the multidimensional descriptive vector comprises an RGB component.

10. The method of claim 1 , further comprising converting the RGB component into a feature vector.

11. The method of claim 10 , wherein the feature vector is a 512-feature vector for a resnet34 deep-learning neural network.

12. The method of claim 1 , wherein each of the feature map patches comprises a fixed-size patch.

13. The method of claim 1 , wherein the feature map patches comprise one of (16, 16, N) or (32, 32, N) tensors, and wherein N is a feature vector size.

14. The method of claim 1 , further comprising generating the fixed-size feature map based on a randomly selected subset of the feature map patches.

15. The method of claim 1 , wherein the subset of the feature map patches is arranged randomly within the fixed-size feature map.

16. The method of claim 1 , further comprising selecting the subset of the feature map patches for further processing.

17. The method of claim 16 , wherein the subset of the feature map patches is randomly selected to define cancer-enriched areas.

18. The method of claim 16 , wherein the subset of the feature map patches is selected to summarize tumor content within a training WSI.

19. The method of claim 1 , wherein the fixed-size feature map comprises one of a (256, 256, N) or (224, 224, N) feature map.

20. The method of claim 1 , wherein the classifier model comprises a modified resnet34 deep-learning neural network.

21. The method of claim 1 , wherein the classifier model comprises a two-layer convolutional deep-learning neural network.

22. The method of claim 1 , wherein the classifier model comprises at least one of an Inception-v3, resnet34, resnet152, densenet169, densenet201 or other deep-learning neural network.

23. The method of claim 1 , wherein each of the plurality of training WSIs corresponds to a different patient.

24. The method of claim 1 , further comprising:
obtaining the test WSI;
generating a varied-size feature map for the test WSIs by generating a grid of patches for the test WSI, segmenting the test WSI into tissue and non-tissue areas, and converting patches comprising the tissue areas into tensors;
generating at least one bounding box based on the patches comprising the tissue areas;
segmenting the at least one bounding box into feature map patches;
generating a fixed-size feature map based on at least a subset of the feature map patches; and
processing the fixed-size feature map using the trained classifier model, wherein the trained classifier model is operable to determine a WSI-level tissue or cell morphology classification or regression for the test WSI based on the fixed-size feature map.

25. An apparatus for determining classifications based on whole slide images (WSIs), the apparatus comprising:
a processor;
a memory device storing software instructions for determining molecular subtype classifications; and
a training engine executable on the processor according to software instructions stored in the memory device and configured to:
obtain a plurality of training WSIs;
generate a varied-size feature map for each of the plurality of training WSIs by generating a grid of patches for the training WSI, segmenting the training WSI into tissue and non-tissue areas, and converting patches comprising the tissue areas into tensors;
generate at least one bounding box based on the patches comprising the tissue areas;
segment the at least one bounding box into feature map patches;
generate a fixed-size feature map based on at least a subset of the feature map patches;
configure a classifier model to process fixed-size feature maps corresponding to the training WSIs such that, for each fixed-size feature map, the classifier model is operable to assign a WSI-level tissue or cell morphology classification or regression based on the tensors;
train the classifier model using the fixed-size feature map corresponding to the plurality of training WSIs; and
configure a classification engine to use the trained classifier model to determine a WSI-level tissue or cell morphology classification or regression for a test WSI.

26. A non-transitory computer-readable medium having computer instructions stored thereon for determining classifications based on whole slide images (WSIs), which, when executed by a processor, cause the processor to perform one or more steps comprising:
obtaining a plurality of training WSIs;
generating a varied-size feature map for each of the plurality of training WSIs by generating a grid of patches for the training WSI, segmenting the training WSI into tissue and non-tissue areas, and converting patches comprising the tissue areas into tensors;
generating at least one bounding box based on the patches comprising the tissue areas;
segmenting the at least one bounding box into feature map patches;
generating a fixed-size feature map based on at least a subset of the feature map patches;
conf

Assignees

• Nantomics Llc
• Nanthealth Inc

Inventors

• Song Bing
• Jaber Mustafa
• Beziaeva Liudmila
• Rabizadeh Shahrooz

Classifications

• G06N3/0895

  Weakly supervised learning, e.g. semi-supervised or self-supervised learning · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/08 Primary

  Learning methods · CPC title

• G06T7/0014

  using an image reference approach · CPC title

• G06T7/194

  involving foreground-background segmentation · CPC title