Determining biomarkers from histopathology slide images

Patent metadata

Field	Value
Publication number	US-11935152-B2
Application number	US-202318123959-A
Country	US
Kind code	B2
Filing date	Mar 20, 2023
Priority date	May 14, 2018
Publication date	Mar 19, 2024
Grant date	Mar 19, 2024

Abstract

A system for identifying biomarkers in a digital image of a Hematoxylin and Eosin-stained slide of a target tissue includes a processor and an electronic network; and a memory having stored thereon computer-executable instructions that, when executed by the one or more processors, cause the computing system to: process segmented tile images determine a predicted biomarker presence; and transmit the predicted presence. A non-transitory computer-readable medium includes a set of computer-executable instructions that, when executed by one or more processors, cause a computer to: process segmented tile images; determine a predicted biomarker presence; and transmit the predicted presence. A computer-implemented method includes processing segmented tile images; determining a predicted biomarker presence; and transmitting the predicted presence.

First claim

What is claimed:

1. A computing system for identifying biomarkers in a digital image of a Hematoxylin and Eosin-stained slide of a target tissue, comprising:
one or more processors;
an electronic network; and
one or more memories having stored thereon computer-executable instructions that, when executed by the one or more processors, cause the computing system to:
process a plurality of segmented tile images each corresponding to a different respective portion of the digital image using a deep learning framework by:
(i) predicting a respective biomarker classification for each tile image using one or more trained biomarker classification models; and
(ii) predicting a respective tissue classification for each tile image using one or more trained deep learning classifier models;
determine, based on (i) and (ii), a predicted presence of one or more biomarkers in the target tissue;
transmit, via the electronic network, the predicted presence of the one or more biomarkers;
receive a molecular training dataset for a plurality of training tissue samples, the molecular training dataset comprising molecular data based on sequencing of a substantially similar sample associated with each training tissue sample; and
identify one or more molecular data subsets in the molecular training dataset, each corresponding to a different respective biomarker, by processing the molecular training dataset using a clustering algorithm.

2. The computing system of claim 1 , the one or more memories having stored thereon further instructions that, when executed by the one or more processors, cause the computing system to:
separate the digital image into the plurality of segmented tile images by processing the digital image using at least one of (i) a tiling mask or (ii) a trained multiple instance learning controller.

3. The computing system of claim 1 , the one or more memories having stored thereon further instructions that, when executed by the one or more processors, cause the computing system to:
receive, at the deep learning framework, at least one training Hematoxylin and Eosin-stained slide image having a respective label corresponding to a respective biomarker;
classify the Hematoxylin and Eosin-stained slide image using tile-based tissue classification analysis; and
analyzing the Hematoxylin and Eosin-stained slide image using a pixel-based cell segmentation.

4. The computing system of claim 3 , the one or more memories having stored thereon further instructions that, when executed by the one or more processors, cause the computing system to:
identify a plurality of cells within the plurality of tile images using a trained cell segmentation model by:
applying each of the plurality of tile images to a cell segmentation model and, for each tile image, assigning a cell classification to one or more pixels within the tile image.

5. The computing system of claim 4 , the one or more memories having stored thereon further instructions that, when executed by the one or more processors, cause the computing system to:
assign the cell classification to one or more pixels within the tile image by:
identifying the one or more pixels as a cell interior, a cell border, or a cell exterior; and
classifying the one or more pixels as the cell interior, the cell border, or the cell exterior.

6. The computing system of claim 4 , wherein the trained cell segmentation model is a pixel-resolution three-dimensional classification model trained to classify a cell interior, a cell border, and a cell exterior.

7. The computing system of claim 3 , the one or more memories having stored thereon further instructions that, when executed by the one or more processors, cause the computing system to:
classify the Hematoxylin and Eosin-stained image using tile-based biomarker classification analysis.

8. The computing system of claim 3 , the one or more memories having stored thereon further instructions that, when executed by the one or more processors, cause the computing system to:
generate one or both of (i) the trained biomarker classification models, and (ii) the trained deep learning classifier models.

9. The computing system of claim 1 , the one or more memories having stored thereon further instructions that, when executed by the one or more processors, cause the computing system to:
for each tile image in the plurality of tile images:
infer a class status of the tile image; and

discard, when the class status of the tile image does not correspond to a desired class, the tile image.

10. The computing system of claim 1 ,
wherein at least one of the trained deep learning classifier models is a tile-resolution Fully Convolutional Network (FCN) classification model.

11. The computing system of claim 1 , wherein the one or more biomarkers include at least one of a tumor-infiltrating lymphocyte (TIL) biomarker, a nucleus-to-cytoplasm (NC) ratio biomarker, a ploidy biomarker, a signet ring morphology biomarker, a programmed death-ligand 1 (PD-L1) biomarker, a consensus molecular subtype (CMS) biomarker, a human epidermal growth factor receptor 2 (HER2) biomarker, or a homologous recombination deficiency (HRD) biomarker.

12. The computing system of claim 1 , wherein the deep learning framework includes at least one of a multi-scale deep learning framework or a single-scale deep learning framework.

13. The computing system of claim 12 , wherein the single-scale deep learning framework is a convolution neural network having a ResNet configuration or an Inception configuration.

14. The computing system of claim 1 , the one or more memories having stored thereon further instructions that, when executed by the one or more processors, cause the computing system to:
for each tile image in the plurality of tile images:
process the tile image using a biomarker classification model trained to predict a different respective biomarker classification; and

determine, based on the predicted biomarkers of the tile image, a predicted presence of one or more biomarkers in the target tissue; and
generate a report containing the digital image and a digital overlay visualizing the predicted presence of the one or more biomarkers.

15. The computing system of claim 14 , wherein the digital overlay includes an overlay element identifying tumor content of the digital image or tumor percentage of the digital image.

16. The computing system of claim 1 , the one or more memories having stored thereon further instructions that, when executed by the one or more processors, cause the computing system to:
for each molecular data subset in the one or more molecular data subsets:
receive a plurality of digital images of Hematoxylin and Eosin-stained training slides of training tissue samples corresponding to the respective different biomarker of the molecular data subset in an image-based biomarker prediction system having one or more processors; and
generate one of the trained biomarker classification models, based on the plurality of digital images of the Hematoxylin and Eosin-stained training slides.

17. The computing system of claim 1 , wherein the computing system further comprises:
a pathology slide scanner system;
and the one or more memories have stored thereon further instructions that, when executed by the one or more processors, cause the computing system to:
receive, via the electronic network, the digital image from the pathology slide scanner system.

18. A non-transitory computer-readable medium comprising a set of computer-executable instructions that, when executed by one or

Assignees

• Tempus Labs Inc

Inventors

• Yip Stephen
• Ho Irvin
• Sha Lingdao
• Osinski Boleslaw
• Khan Aly Azeem
• Kruger Andrew J
• Carlson Michael
• Greenwald Abel
• Willis Caleb

Classifications

• G06N3/0895

  Weakly supervised learning, e.g. semi-supervised or self-supervised learning · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06T1/20 Primary

  Processor architectures; Processor configuration, e.g. pipelining · CPC title

• G06F18/21

  Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation · CPC title