Determining biomarkers from histopathology slide images

Patent metadata

Field	Value
Publication number	US-11610307-B2
Application number	US-202117192383-A
Country	US
Kind code	B2
Filing date	Mar 4, 2021
Priority date	May 14, 2018
Publication date	Mar 21, 2023
Grant date	Mar 21, 2023

Abstract

A generalizable and interpretable deep learning model for predicting biomarker status and biomarker metrics from histopathology slide images is provided.

First claim

What is claimed:

1. A computing device configured to identifying biomarkers in digital image of a Hematoxylin and eosin (H&E) stained slide of target tissue, the computing device comprising:
one or more memories; and
one or more processors configured to,
receive the digital image to an image-based biomarker prediction system having one or more processors;
separate the digital image into a plurality of tile images, where each of the plurality of tile images contains a different portion of the digital image;
apply the plurality of tile images to a deep learning framework comprising one or more trained biomarker classification models, each trained biomarker classification model being trained to classify a different biomarker, wherein the deep learning framework comprises a multiscale deep learning framework;
predict a biomarker classification for each of the plurality of tile images using the one or more trained biomarker classification models;
from the predicted biomarker classifications of each of the tile images, determine a predicted presence of one or more biomarkers in the target tissue; and
generate a report containing the digital image and a digital overlay visualizing the predicted presence of the one or more biomarkers,
wherein to apply the plurality of tile images to the deep learning framework and to predict the biomarker classification for each of the plurality of tile images, the one or more processors are further configured to,
apply each of the tile images to one or more trained deep learning multiscale classifier models, each trained deep learning multiscale classifier model being trained to classify a different tissue classification for each tile image and determine a tissue classification for each of the plurality of tile images, using the multiscale deep learning framework,
identify, using the one or more processors, cells within the digital image using a trained cell segmentation model, and
from the tissue classification determined for each tile image and from the identified cells within the digital image, predict the biomarker classification for each tile image.

2. The computing device of claim 1 , wherein the one or more processors are configured to separate the digital image into a plurality of tile images by:
performing an image tiling process, using the one or more processors, by applying a tiling mask to the digital image to separate the digital image into the plurality of tile images.

3. The computing device of claim 1 , wherein the tiling mask comprises tiles of the same size.

4. The computing device of claim 1 , wherein the tiling mask comprises tiles having a rectangular shape.

5. The computing device of claim 1 , wherein the one or more processors are further configured to train the one or more trained deep learning multiscale classifier models by:
receiving, at the multiscale deep learning framework, a plurality of H&E slide training images from a training images dataset, each H&E slide training image having a label corresponding to a biomarker to be trained;
performing tile-based tissue classification analysis on each of the H&E slide training images;
performing a pixel-based cell segmentation analysis on each of the H&E slide training images;
optionally performing a tile-based biomarker classification analysis on each of the H&E slide training images; and
in response, generating the one or more trained deep learning multiscale classifier models.

6. The computing device of claim 1 , wherein the one or more processors are further configured to:
for each H&E slide training image, perform a tile selection process that infers a class status for each tile image in the H&E slide training image;
based on inferred class status, discard tile images not corresponding to a desired class, before performing the tile-based tissue classification analysis on each of the H&E slide training images, such that the tile-based tissue classification analysis is performed on only selected tile images of the H&E slide training image.

7. The computing device of claim 1 , wherein the one of the one or more trained deep learning multiscale classifier models are each configured as a tile-resolution Fully Convolutional Network (FCN) classification model.

8. The computing device of claim 1 , wherein the one or more biomarkers are selected from the group consisting of tumor-infiltrating lymphocytes (TILs), nucleus-to-cytyoplasm (NC) ratio, ploidy, signet ring morphology, and programmed death-ligand 1 (PD-L1).

9. The computing device of claim 1 , wherein the one or more biomarkers are selected from the group consisting of consensus molecular subtype (CMS) and homologous recombination deficiency (“HRD”).

10. The computing device of claim 1 , wherein the one or more processors are one or more graphics processing units (GPUs), tensor processing units (TPUs), and/or central processing units (CPUs).

11. The computing device of claim 1 , wherein the computing device is communicatively coupled to a pathology slide scanner system through a communication network, such that the computing device receives the digital image from the pathology slide scanner system over the communication network.

12. The computing device of claim 1 , wherein the computing device is contained within a pathology slide scanner system.

13. The computing device of claim 1 , wherein at least one of the one or more processors of the computing device is contained within a pathology slide scanner system.

14. The computing device of claim 1 , wherein the one or more processors are configured to generate the report containing the digital image and the digital overlay by generating the digital overlay to include an overlay element identifying tumor content of the digital image or tumor percentage of the digital image.

15. The computing device of claim 1 , wherein the one or more processors are configured to identify cells within the digital image tile using the trained cell segmentation model by:
applying, using the one or more processors, each of the plurality of tile images to the cell segmentation model and, for each tile, assigning a cell classification to one or more pixels within the tile image.

16. The computing device of claim 15 , wherein the one or more processors are configured to assign the cell classification to one or more pixels within the tile image by:
identifying, using the one or more processors, the one or more pixels as a cell interior, a cell border, or a cell exterior and classifying the one or more pixels as the cell interior, the cell border, or the cell exterior.

17. The computing device of claim 1 , wherein the trained cell segmentation model is a pixel-resolution three-dimensional UNet classification model trained to classify a cell interior, a cell border, and a cell exterior.

18. The computing device of claim 1 , wherein the one or more biomarkers comprises a tumor-infiltrating lymphocytes (TILs) biomarker.

19. The computing device of claim 1 , wherein the one or more biomarkers comprises a nucleus-to-cytyoplasm (NC) ratio biomarker.

20. The computing device of claim 1 , wherein the one or more biomarkers comprises a human epidermal growth factor receptor 2 (HER2) biomarker.

21. The computing device of claim 1 , wherein the one or more biomarkers comprises a programmed death-ligand 1 (PD-L1) biomarker.

22. A computing device configured to identifying biomarkers in digital image of a Hematoxylin and eosin (H&E) stained slide of target tissue, the computing device compri

Assignees

• Tempus Labs Inc

Inventors

• Yip Stephen
• Ho Irvin
• Sha Lingdao
• Osinski Boleslaw
• Khan Aly Azeem
• Kruger Andrew J
• Carlson Michael
• Greenwald Abel
• Willis Caleb

Classifications

• G06N3/0895

  Weakly supervised learning, e.g. semi-supervised or self-supervised learning · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06T2207/30096

  Tumor; Lesion · CPC title

• G06T2207/10024

  Color image · CPC title