Determining biomarkers from histopathology slide images

Patent metadata

Field	Value
Publication number	US-10957041-B2
Application number	US-202016830186-A
Country	US
Kind code	B2
Filing date	Mar 25, 2020
Priority date	May 14, 2018
Publication date	Mar 23, 2021
Grant date	Mar 23, 2021

Abstract

A generalizable and interpretable deep learning model for predicting biomarker status and biomarker metrics from histopathology slide images is provided.

First claim

What is claimed:

1. A computer-implemented method of identifying biomarkers in a digital image of a hematoxylin and eosin (H&E) stained slide of target tissue, the method comprising:
receiving the digital image to an image-based biomarker prediction system having one or more processors;
separating, using the one or more processors, the digital image into a plurality of tile images, where each of the plurality of tile images contains a different portion of the digital image;
applying, using the one or more processors, the plurality of tile images to a deep learning framework comprising one or more trained biomarker classification models, each trained biomarker classification model being trained to classify a different biomarker, wherein the deep learning framework comprises a multiscale deep learning framework;
predicting, using the one or more processors, a biomarker classification for each of the plurality of tile images using the one or more trained biomarker classification models;
from the predicted biomarker classifications of each of the tile images, determining a predicted presence of one or more biomarkers in the target tissue; and
generating a report containing the digital image and a digital overlay visualizing the predicted presence of the one or more biomarkers,
wherein each of the applying the plurality of tile images to the deep learning framework and predicting the biomarker classification for each of the plurality of tile images comprises,
applying each of the tile images to one or more trained deep learning multiscale classifier models, each trained deep learning multiscale classifier models being trained to classify a different tissue classification for each tile image and determining a tissue classification for each of the plurality of tile images, using the multiscale deep learning framework,
identifying, using the one or more processors, cells within the digital image using a trained cell segmentation model, and
from the tissue classification determined for each tile image and from the identified cells within the digital image, predicting the biomarker classification for each tile image.

2. The method of claim 1 , wherein separating the digital image into a plurality of tile images comprises:
performing an image tiling process, using the one or more processors, by applying a tiling mask to the digital image to separate the digital image into the plurality of tile images.

3. The method of claim 2 , wherein the tiling mask comprises tiles of the same size.

4. The method of claim 3 , wherein the tiling mask comprises tiles having a rectangular shape.

5. The method of claim 1 , further comprising training the one or more trained deep learning multiscale classifier models by:
receiving, at the multiscale deep learning framework, a plurality of H&E slide training images from a training images dataset, each H&E slide training image having a label corresponding to a biomarker to be trained;
performing tile-based tissue classification analysis on each of the H&E slide training images;
performing a pixel-based cell segmentation analysis on each of the H&E slide training images;
optionally performing a tile-based biomarker classification analysis on each of the H&E slide training images; and
in response, generating the one or more trained deep learning multiscale classifier models.

6. The method of claim 5 , wherein each H&E slide training image comprises a plurality of tile images each having a tile-level label.

7. The method of claim 5 , further comprising, for each H&E slide training image, imputing a tile-level label for each of a plurality of tile images of the H&E slide training image.

8. The method of claim 5 , further comprising:
for each H&E slide training image, performing a tile selection process that infers a class status for each tile image in the H&E slide training image; and
based on inferred class status, discarding tile images not corresponding to a desired class, before performing the tile-based tissue classification analysis on each of the H&E slide training images, such that the tile-based tissue classification analysis is performed on only selected tile images of the H&E slide training image.

9. The method of claim 1 , wherein the one of the one or more trained deep learning multiscale classifier models are each configured as a tile-resolution Fully Convolutional Network (FCN) classification model.

10. The method of claim 1 , wherein identifying cells within the digital image tile using the trained cell segmentation model comprises:
applying, using the one or more processors, each of the plurality of tile images to the cell segmentation model and, for each tile, assigning a cell classification to one or more pixels within the tile image.

11. The method of claim 10 , wherein assigning the cell classification to one or more pixels within the tile image comprises:
identifying, using the one or more processors, the one or more pixels as a cell interior, a cell border, or a cell exterior and classifying the one or more pixels as the cell interior, the cell border, or the cell exterior.

12. The method of claim 1 , wherein the trained cell segmentation model is a pixel-resolution three-dimensional UNet classification model trained to classify a cell interior, a cell border, and a cell exterior.

13. The method of claim 1 , wherein the one or more biomarkers are selected from the group consisting of tumor-infiltrating lymphocytes (TILs), nucleus-to-cytyoplasm (NC) ratio, ploidy, signet ring morphology, and programmed death-ligand 1 (PD-L1).

14. The method of claim 1 , wherein the one or more processors are one or more graphics processing units (GPUs), tensor processing units (TPUs), and/or central processing units (CPUs).

15. The method of claim 1 , wherein the image-based biomarker prediction system is communicatively coupled to a pathology slide scanner system through a communication network, such that the image-based biomarker prediction system receives the digital image from the pathology slide scanner system over the communication network.

16. The method of claim 1 , wherein the image-based biomarker prediction system is contained within a pathology slide scanner system.

17. The method of claim 16 , wherein the pathology slide scanner system contains an image-based, adversarial trained, and/or a microsatellite instability (MSI) prediction model.

18. The method of claim 1 , wherein generating the report containing the digital image and the digital overlay comprises generating the digital overlay to include an overlay element identifying tumor content of the digital image or tumor percentage of the digital image.

19. The method of claim 1 , wherein the one or more biomarkers comprises a tumor-infiltrating lymphocytes (TILs) biomarker.

20. The method of claim 1 , wherein the one or more biomarkers comprises a nucleus-to-cytyoplasm (NC) ratio biomarker.

21. The method of claim 1 , wherein the one or more biomarkers comprises a human epidermal growth factor receptor 2 (HER2) biomarker.

22. The method of claim 1 , wherein the one or more biomarkers comprises a programmed death-ligand 1 (PD-L1) biomarker.

23. A computer-implemented method of identifying biomarkers in a digital image of a hematoxylin and eosin (H&E) stained slide of target tissue, the method comprising:
receiving the digital image to an image-based biomarker prediction system having one or more processors;
separating, using the one or more processor

Assignees

• Tempus Labs Inc

Inventors

• Yip Stephen
• Ho Irvin
• Sha Lingdao
• Osinski Boleslaw
• Khan Aly Azeem
• Kruger Andrew J
• Carlson Michael
• Greenwald Abel
• Willis Caleb

Classifications

• G06V10/44

  Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components · CPC title

• G06V10/82

  using neural networks · CPC title

• G06V10/764

  using classification, e.g. of video objects · CPC title

• G06T7/0012 Primary

  Biomedical image inspection · CPC title

• G06T1/20 Primary

  Processor architectures; Processor configuration, e.g. pipelining · CPC title