Multi-resolution segmentation for gigapixel images

Patent metadata

Field	Value
Publication number	US-12354262-B2
Application number	US-202218082406-A
Country	US
Kind code	B2
Filing date	Dec 15, 2022
Priority date	Jan 31, 2022
Publication date	Jul 8, 2025
Grant date	Jul 8, 2025

Abstract

Systems and methods for determining pixel classification information using images depicting at least a portion of a whole slide image (WSI) of a stained tissue sample. A system can store a first image of the tissue sample at a first resolution, a second image of the tissue sample at a second resolution that is higher than the first resolution, and a third image of the tissue sample at a third resolution that is higher than the second resolution, the first, second, and third images depicting at least a portion of a same area of the tissue sample. The system can include be configured to generate first feature information based on the first image, generate second feature information based on the second image, and determine pixel classification information of at least a portion of the WSI based on the third image, the first feature information and second feature information.

First claim

What is claimed is:

1. An apparatus for determining pixel classification information using a set of images depicting at least some of a whole slide image (WSI) of a stained tissue sample, comprising:
a non-transitory computer storage medium configured to store executable instructions, and
one or more hardware processors in communication with the computer storage medium, wherein the executable instructions, when executed by the one or more hardware processors, configure the one or more hardware processors to
generate first feature information based on a first context image of the tissue sample at a first resolution;
generate additional feature information n separately for one or more additional context images n where n is an integer, wherein when n=1 the additional context image 1 has a resolution 1 which is higher than the first resolution, wherein when n>1 each additional context image n has a resolution n which is higher than the resolution of the additional context image n−1, and wherein the first context image, target image, and each additional context image n depicts at least a portion of a same area of the tissue sample; and
determine pixel classification information of at least a portion of the WSI based on a target image of the tissue sample at a second resolution that is higher than the highest resolution of any of the one or more additional context images n, the first feature information, and each additional feature information n.

2. The apparatus of claim 1 , wherein the executable instructions, when executed by the one or more hardware processors, further configure the one or more hardware processors to:
generate the first feature information using a first encoder-decoder trained for feature detection and pixel classification using an enriched dataset including images of the first resolution, wherein the first feature information is an intermediary output of the first encoder-decoder,
generate each of the additional feature information n using a encoder-decoder n trained for feature detection and pixel classification using an enriched dataset including images of the resolution n, wherein the additional feature information n is an intermediary output of the encoder-decoder n,
determine the pixel classification information of at least a portion of the WSI based on the target image using a target encoder-decoder trained for feature detection and pixel classification using an enriched data set that includes target images.

3. The apparatus of claim 2 , wherein the executable instructions, when executed by the one or more hardware processors, configure the one or more hardware processors to:
generate the first feature information prior to generating a final output from the first encoder-decoder, and
for each additional feature information n, generating the additional feature information n prior to generating a final output of a pixel classification from each encoder-decoder n.

4. The apparatus of claim 2 , wherein each encoder-decoder includes multiple encoder layers and decoder layers.

5. The apparatus of claim 4 , wherein the number of encoder layers of at least two of the encoder-decoders are different.

6. The apparatus of claim 4 , wherein the number of decoder layers of at least two of the encoder-decoders are different.

7. The apparatus of claim 2 , wherein the executable instructions, when executed by the one or more hardware processors, further configure the one or more hardware processors to store the first, and additional feature information n, and provide the first, and additional feature information n to the target encoder-decoder.

8. The apparatus of claim 2 , where the target encoder-decoder includes a layer that processes image data having the same resolution as the resolution of the first context image, and each of the additional context images n.

9. The apparatus of claim 2 , wherein each encoder-decoder includes a convolutional neural network.

10. The apparatus of claim 9 , wherein each encoder-decoder includes a convolutional neural network trained using an enriched dataset including images of the same resolution as the images the encoder-decoder is used to inference.

11. The apparatus of claim 9 , wherein each encoder-decoder includes a convolution neural network trained using an enriched dataset including only images of the same resolution as the images the encoder-decoder is used to inference.

12. The apparatus of claim 1 , wherein the first, and n th images depict spatially concentric areas of the tissue sample, or spatially non-concentric areas of the tissue sample.

13. The apparatus of claim 1 , wherein the first, second and n th images depict spatially non-concentric areas of the tissue sample.

14. The apparatus of claim 1 , wherein:
the first context image is comprised by a set of first context images, wherein:
each image from the set of first context images depicts a different portion of the tissue sample at the first resolution; and
each of the one or more additional context images n is comprised by a set of context images n wherein:
for each set of context images n, each image in that set has the resolution n;
the set of context images 1 comprises a plurality of subsets of context images 1, wherein each subset from the plurality of subsets of context images 1 corresponds to a different image from the set of first context images;
for each subset from the plurality of subsets of context images 1, each image in that subset depicts a different portion of the portion of the tissue sample depicted in the image from the set of first context images which corresponds to that subset of context images 1;
when n>1:
the set of context images n comprises a plurality of subsets of context images n, wherein each subset from the plurality of subsets of context images n corresponds to a different image from the set of context images n−1; and
for each subset from the plurality of subsets of context images n, each image in that subset depicts a different portion of the portion of the tissue sample depicted in the image from the set of context images n−1 which corresponds to that subset of context images n;
the target image of the tissue sample is comprised by a set of target images, wherein:
the set of target images comprises a plurality of subsets of target images, wherein each subset from the plurality of subsets of target images corresponds to a different image from the set of context images n;
for each subset from the plurality of subsets of target images, each image in that subset depicts a different portion of the portion of the tissue sample depicted in the image from the set of context images n which corresponds to that subset of target images; and
for each image from the set of target images, the portion of the tissue sample depicted in that image is depicted in that image at the second resolution;
the executable instructions, when executed by the one or more hardware processors, configure the one or more hardware processors to:
for each subset of context images 1, generate first feature information based on the image from the set of first context images which corresponds to that subset before generating additional feature information 1 based on any of the images from that subset;
when n>1, for each subset of context images n, generate additional feature information n−1 based on the image from the from the set of context images n−1 which corresponds to that subset before generating additional feature information n based on any of the images from that subset;
for each subset of target images, before determining pixel classification information based on any image from that subset, gene

Assignees

• Leica Biosystems Imaging Inc

Inventors

• Miller Derek
• Salinas Chad

Classifications

• G06T2207/30024

  Cell structures in vitro; Tissue sections in vitro · CPC title

• G06T2207/20016

  Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform · CPC title

• G06T3/4053

  based on super-resolution, i.e. the output image resolution being higher than the sensor resolution · CPC title

• G06V10/764

  using classification, e.g. of video objects · CPC title

• G06V10/44

  Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components · CPC title