High and low frequency feature map generation for H and E pathology images

Patent metadata

Field	Value
Publication number	US-12475684-B2
Application number	US-202318115082-A
Country	US
Kind code	B2
Filing date	Feb 28, 2023
Priority date	Feb 28, 2022
Publication date	Nov 18, 2025
Grant date	Nov 18, 2025

Abstract

An apparatus and methods for determining features in an image of a Hematoxylin and Eosin (H&E) stained tissue sample. An apparatus can be configured to apply a machine learning model to the H&E stained tissue sample image to determine features in the image. Applying the machine learning model can include performing convolution operations on the H&E stained tissue sample image to generate a plurality of initial feature maps, applying octave-convolution-first-layer operations on the initial feature maps to generate initial high-frequency feature maps and low-frequency feature maps, applying octave-convolution operations on the high-frequency initial feature maps to generate refined high-frequency feature maps, applying octave-convolution operations on the low-frequency initial feature maps to generate refined low-frequency feature maps, and applying octave-convolution last layer operations on the refined high-frequency feature maps and the refined low-frequency feature maps to produce combined feature maps.

First claim

What is claimed is:

1 . An apparatus for determining features in an image of a Hematoxylin and Eosin (H&E) stained tissue sample, comprising:
a non-transitory computer readable medium configured to store executable instructions, and to store a H&E stained tissue sample image; one or more hardware processors in communication with the computer storage medium, wherein the executable instructions, when executed by the one or more hardware processors, configure the one or more hardware processors to:
apply a machine learning model to the H&E stained tissue sample image, the machine learning model including at least a feature extractor network, wherein applying the machine learning process includes at least:
performing convolution operations on the H&E stained tissue sample image to generate a plurality of initial feature maps;
applying octave-convolution-first-layer operations on the initial feature maps to generate initial high-frequency feature maps and low-frequency feature maps;
applying octave-convolution operations on the high-frequency initial feature maps to generate refined high-frequency feature maps;
applying octave-convolution operations on the low-frequency initial feature maps to generate refined low-frequency feature maps; and
applying octave-convolution last layer operations on the refined high-frequency feature maps and the refined low-frequency feature maps to produce combined feature maps.

2 . The system of claim 1 , further comprising determining classifications of the features in the combined feature maps.

3 . The system of claim 2 , wherein the classification comprises semantic segmentation.

4 . The system of claim 1 , wherein the spatial resolution of the initial low-frequency feature maps is ½ the spatial resolution of the initial high-frequency feature maps.

5 . The system of claim 1 , wherein the method further comprises performing pooling operations to generate the low-frequency feature maps.

6 . The system of claim 1 , wherein applying octave-convolution last layer operations on the refined high-frequency feature maps and the refined low-frequency feature maps to produce combined feature maps comprises pooling information from the refined high-frequency feature maps and the refined low-frequency feature maps to produce combined feature maps.

7 . The system of claim 1 , wherein applying octave-convolution last layer operations comprises exchanging information between the refined high-frequency feature maps and the refined low-frequency feature maps by pooling and upsampling.

8 . The system of claim 1 , wherein applying octave-convolution last layer operations on the refined high-frequency feature maps and the refined low-frequency feature maps to produce combined feature maps comprises providing information from at least one refined high-frequency feature map to a refined low-frequency feature map by pooling.

9 . The system of claim 1 , wherein applying octave-convolution last layer operations on the refined high-frequency feature maps and the refined low-frequency feature maps to produce combined feature maps comprises communicating information from at least one refined low-frequency feature map to a refined high-frequency feature map by upsampling.

10 . The system of claim 1 , wherein the machine learning process is implemented using a machine learning model comprising a convolutional neural network.

11 . A method for determining features in an image of a Hematoxylin and Eosin (H&E) stained tissue sample using a machine learning model including at least a feature extractor network, the method comprising:
performing convolution operations on the H&E stained tissue sample image to generate a plurality of initial feature maps; applying octave-convolution-first-layer operations on the initial feature maps to generate initial high-frequency feature maps and low-frequency feature maps; applying octave-convolution operations on the high-frequency initial feature maps to generate refined high-frequency feature maps; applying octave-convolution operations on the low-frequency initial feature maps to generate refined low-frequency feature maps; and applying octave-convolution last layer operations on the refined high-frequency feature maps and the refined low-frequency feature maps to produce combined feature maps, wherein the method is performed by one or more computer hardware processors executing program instructions stored on a non-transitory computer medium.

12 . The method of claim 11 , further comprises determining classifications of the features in the combined feature maps.

13 . The method of claim 11 , wherein the classification comprises semantic segmentation.

14 . The method of claim 11 , wherein the spatial resolution of the initial low-frequency feature maps is ½ the spatial resolution of the initial high-frequency feature maps.

15 . The method of claim 11 , wherein applying octave-convolution-first-layer operations on the initial feature maps to generate initial high-frequency feature maps and low-frequency feature maps comprises performing pooling operations to generate the low-frequency feature maps.

16 . The method of claim 11 , wherein applying octave-convolution last layer operations on the refined high-frequency feature maps and the refined low-frequency feature maps to produce combined feature maps comprises pooling information from the refined high-frequency feature maps and the refined low-frequency feature maps to produce combined feature maps.

17 . The method of claim 11 , wherein applying octave-convolution last layer operations comprises exchanging information between the refined high-frequency feature maps and the refined low-frequency feature maps by pooling and upsampling.

18 . The method of claim 11 , wherein applying octave-convolution last layer operations on the refined high-frequency feature maps and the refined low-frequency feature maps to produce combined feature maps comprises providing information from at least one refined high-frequency feature map to a refined low-frequency feature map by pooling.

19 . The method of claim 11 , wherein applying octave-convolution last layer operations on the refined high-frequency feature maps and the refined low-frequency feature maps to produce combined feature maps comprises communicating information from at least one refined low-frequency feature map to a refined high-frequency feature map by upsampling.

20 . The method of claim 11 , wherein the H&E stained tissue sample image is an image of a slide from a tissue block.

Assignees

• Leica Biosystems Imaging Inc

Inventors

• Wang Ji
• Salinas Chad

Classifications

• G06T2207/30024

  Cell structures in vitro; Tissue sections in vitro · CPC title

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title

• G06T7/0012

  Biomedical image inspection · CPC title

• G06T3/40

  Scaling of whole images or parts thereof, e.g. expanding or contracting · CPC title

• G06V10/764

  using classification, e.g. of video objects · CPC title