Machine-learned hormone status prediction from image analysis

Patent metadata

Field	Value
Publication number	US-11508481-B2
Application number	US-202016895983-A
Country	US
Kind code	B2
Filing date	Jun 8, 2020
Priority date	Mar 6, 2020
Publication date	Nov 22, 2022
Grant date	Nov 22, 2022

Abstract

An analytics system uses one or more machine-learned models to predict a hormone receptor status from a H&E stain image. The system partitions H&E stain images each into a plurality of image tiles. Bags of tiles are created through sampling of the image tiles. The analytics system trains one or more machine-learned models with training H&E stain images having a positive or negative receptor status. The analytics system generates, via a tile featurization model, a tile feature vector for each image tile a test bag for a test H&E stain image. The analytics system generates, via an attention model, an aggregate feature vector for the test bag by aggregating the tile feature vectors of the test bag, wherein an attention weight is determined for each tile feature vector. The analytics system predicts a hormone receptor status by applying a prediction model to the aggregate feature vector for the test bag.

First claim

What is claimed is:

1. A method for predicting hormone receptor status using machine learning comprising:
accessing a test haematoxylin and eosin (H&E) stain image of a test tissue sample;
partitioning the test H&E stain image into a plurality of non-overlapping image tiles;
sampling a subset of non-overlapping image tiles from the plurality of non-overlapping image tiles of the test H&E stain image;
generating a tile feature vector for one or more non-overlapping image tiles of the sampled subset of non-overlapping image tiles;
generating an aggregate feature vector for the test H&E stain image by aggregating the tile feature vectors of the non-overlapping image tiles of the sampled subset; and
predicting a hormone receptor status by applying a machine-learned prediction model to the aggregate feature vector for the test H&E stain image, wherein the machine-learned prediction model is trained using a first set of H&E stain images from a first set of tissue samples having a positive hormone receptor status and a second set of H&E stain images from a second set of tissue samples having a negative hormone receptor status.

2. The method of claim 1 , wherein the hormone receptor status comprises an estrogen receptor status.

3. The method of claim 1 , wherein the plurality of image tiles are non-overlapping.

4. The method of claim 1 , further comprising filtering the plurality of non-overlapping image tiles to retain non-overlapping image tiles pertaining to the test tissue sample, wherein the sampled subset of non-overlapping image tiles is sampled from the non-overlapping image tiles pertaining to the test tissue sample.

5. The method of claim 1 , wherein generating the tile feature vector for the one or more non-overlapping image tile comprises applying a tile featurization model to each non-overlapping image tile.

6. The method of claim 5 , wherein generating the aggregate feature vector comprises applying an attention model to the tile feature vectors of the sampled subset of non-overlapping image tiles, wherein the attention model is configured to determine, for each tile feature vector of the sampled subset, an attention weight based on each tile feature vector and to sum products of each attention weight and the respective tile feature vector to produce the aggregate feature vector.

7. The method of claim 6 , wherein the featurization model, the attention model, and the machine-learned prediction model are trained synchronously using the first set of H&E stain images from the first set of tissue samples having the positive hormone receptor status and the second set of H&E stain images from the second set of tissue samples having the negative hormone receptor status.

8. The method of claim 1 , wherein the predicted hormone receptor status is one of:
a positive status or a negative status; and
a likelihood of a positive status or a likelihood of a negative status.

9. The method of claim 1 , further comprising:
sampling a second subset of non-overlapping image tiles from the plurality of non-overlapping image tiles of the test H&E stain image;
generating a tile feature vector for one or more non-overlapping image tiles of the second sampled subset of non-overlapping image tiles;
generating a second aggregate feature vector for the test H&E stain image by aggregating the tile feature vectors of the non-overlapping image tiles of the second sampled subset; and
predicting a second hormone receptor status by applying the machine-learned prediction model to the second aggregate feature vector, wherein a cumulative prediction of the hormone receptor status combines the hormone receptor status and the second hormone receptor status.

10. A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform operations comprising:
accessing a test haematoxylin and eosin (H&E) stain image of a test tissue sample;
partitioning the test H&E stain image into a plurality of non-overlapping image tiles;
sampling a subset of non-overlapping image tiles from the plurality of non-overlapping image tiles for the test H&E stain image;
generating a tile feature vector for one or more non-overlapping image tiles of the sampled subset of image tiles;
generating an aggregate feature vector for the test H&E stain image by aggregating the tile feature vectors of the non-overlapping image tiles in the sampled subset; and
predicting a hormone receptor status by applying a machine-learned prediction model to the aggregate feature vector for the test H&E stain image, wherein the machine-learned prediction model is trained using a first set of H&E stain images from a first set of tissue samples having a positive hormone receptor status and a second set of H&E stain images from a second set of tissue samples having a negative hormone receptor status.

11. The storage medium of claim 10 , the operations further comprising filtering the plurality of non-overlapping image tiles to retain non-overlapping image tiles pertaining to the test tissue sample, wherein the sampled subset of non-overlapping image tiles is sampled from the non-overlapping image tiles pertaining to the test tissue sample.

12. The storage medium of claim 10 ,
wherein generating the tile feature vector for one or more non-overlapping image tiles comprises applying a tile featurization model to each non-overlapping image tile,
wherein generating the aggregate feature vector comprises applying an attention model to the tile feature vectors of the sampled subset of image tiles, wherein the attention model is configured to determine, for each tile feature vector of the sampled subset, an attention weight based on each tile feature vector and to sum products of each attention weight and the respective tile feature vector to produce the aggregate feature vector, and
wherein the featurization model, the attention model, and the machine-learned prediction model are trained synchronously using the first set of H&E stain images from the first set of tissue samples having the positive hormone receptor status and the second set of H&E stain images from the second set of tissue samples having the negative hormone receptor status.

13. The storage medium of claim 10 , the operations further comprising:
sampling a second subset of non-overlapping image tiles from the plurality of non-overlapping image tiles of the test H&E stain image;
generating a tile feature vector for one or more non-overlapping image tiles of the second sampled subset of non-overlapping image tiles;
generating a second aggregate feature vector for the test H&E stain image by aggregating the tile feature vectors of the non-overlapping image tiles of the second sampled subset; and
predicting a second hormone receptor status by applying the machine-learned prediction model to the second aggregate feature vector, wherein a cumulative prediction of the hormone receptor status combines the hormone receptor status and the second hormone receptor status.

14. A method for training one or more machine-learned models configured to predict hormone receptor status comprising:
accessing a first set of H&E stain images from a first set of tissue samples having a positive hormone receptor status and a second set of H&E stain images from a second set of tissue samples having a negative hormone receptor status;
partitioning each H&E stain image in the first set of H&E stain images and the second set of H&E stain images into a plurality of non-overlapping image tiles;
for each partitioned H&E stain image, sampling a subset of non-overlapping image tiles from the plurality

Assignees

• Salesforce Com Inc
• Salesforce Inc

Inventors

• Naik Nikhil
• Madani Ali
• Keskar Nitish Shirish

Classifications

• G06V20/69

  Microscopic objects, e.g. biological cells or cellular parts · CPC title

• G06T7/0012 Primary

  Biomedical image inspection · CPC title

• G16H50/20 Primary

  for computer-aided diagnosis, e.g. based on medical expert systems · CPC title

• G06F18/2148

  characterised by the process organisation or structure, e.g. boosting cascade · CPC title

• G06F18/217

  Validation; Performance evaluation; Active pattern learning techniques · CPC title