Microscopy system and method for instance segmentation

Patent metadata

Field	Value
Publication number	US-12567151-B2
Application number	US-202217954417-A
Country	US
Kind code	B2
Filing date	Sep 28, 2022
Priority date	Oct 1, 2021
Publication date	Mar 3, 2026
Grant date	Mar 3, 2026

Abstract

A computer-implemented method for instance segmentation of at least one microscope image showing a plurality of objects, comprising: calculating positions of object centers of the objects in the microscope image; determining which image areas of the microscope image are covered by the objects; calculating Voronoi regions using the object centers as Voronoi sites; and determining an instance segmentation mask by separating the image areas covered by the objects into different instances using boundaries of the Voronoi regions.

First claim

What is claimed is:

1 . A computer-implemented method for an instance segmentation of at least one microscope image showing a plurality of objects, comprising:
calculating positions of object centers of the objects in the microscope image; determining which image areas of the microscope image are covered by the objects; calculating Voronoi regions using the object centers as Voronoi sites to determine boundaries between objects of a same class which are touching each other; determining an instance segmentation mask at least by separating the image areas covered with the objects of the same class into different instances using boundaries of the Voronoi regions between objects of the same class; ensuring a number of instances in the instance segmentation mask is equal to a number of the Voronoi sites, in a case where boundaries of the Voronoi regions split the image areas covered with the objects such that one of the Voronoi regions forms two separate regions, by discarding one of the two separate regions or by assigning the one of the two separate regions to one or more neighboring Voronoi regions.

2 . The method according to claim 1 ,
wherein the boundaries of the Voronoi regions are calculated such that each boundary has a same distance to the two nearest object centers; wherein the distance is calculated as a Euclidean distance.

3 . The method according to claim 1 ,
wherein a metric differing from a Euclidean metric is defined as a function of pixel values of the at least one microscope image such that a distance between two pixels depends on pixel values of pixels between said two pixels; wherein the boundaries of the Voronoi regions are calculated such that each boundary has a same distance to the two nearest object centers, wherein the distance is determined according to the metric.

4 . The method according to claim 1 ,
wherein the boundaries of the Voronoi regions are calculated such that each boundary has a same distance to the two nearest object centers, wherein the distance is a shortest path weighted by structures in the microscope image.

5 . The method according to claim 4 ,
wherein the objects are biological cells and wherein the structures used for a weighting in the determination of the distance are cell walls.

6 . The method according to claim 1 ,
wherein a confidence map is calculated for the instance segmentation mask, the confidence map indicating a confidence for each pixel or instance of the instance segmentation mask, wherein the confidence map is calculated as a function of distances of the respective pixels from neighboring object centers.

7 . The method according to claim 6 ,
wherein, as the distance of a pixel from the nearest neighboring object center point decreases, the confidence indicated in the confidence map for that pixel increases; or wherein the confidence map indicates a confidence for a pixel that increases in proportion to a magnitude of a difference between a distance of said pixel from the nearest object center and a distance of said pixel from a second nearest object center.

8 . The method according to claim 1 ,
wherein, in a case where one of the Voronoi regions is separated into two separate regions in the separation of the covered image areas using the boundaries of the Voronoi regions, a correction is carried out in which a smaller region of the two separate regions is discarded or assigned to one or more neighboring Voronoi regions.

9 . The method according to claim 1 ,
wherein the object centers and the image areas covered by objects are determined from the same microscope image.

10 . The method according to claim 1 ,
wherein the object centers and the image areas covered by objects are determined from different microscope images, wherein the different microscope images are registered and captured with different microscopy techniques or microscope settings.

11 . The method according to claim 1 ,
wherein the instance segmentation mask is used for an object analysis in one or more of the following manners:
by cutting out image sections of the microscope image containing objects with pixel precision using the instance segmentation mask and feeding these image sections to a subsequent model for an object analysis;
by determining object sizes from the instance segmentation mask;
by tracking objects spatially over a time series;
by calculating morphological features of objects and subsequently filtering objects as a function of the morphological features; or
wherein the instance segmentation mask is used for an interactive verification of data annotations by displaying the instance segmentation mask or a colored representation of the instance segmentation mask to a human for the verification of object centers.

12 . The method according to claim 1 , further comprising:
using the instance segmentation mask to determine errors in the calculated positions of object centers or the image areas covered by objects, wherein an error is inferred as a function of an extent to which a size or shape of an instance of the instance segmentation mask deviates from an average size or shape of the instances of the instance segmentation mask.

13 . The method according to claim 1 ,
wherein respective instance segmentation masks are calculated for a plurality of microscope images; wherein the instance segmentation masks are used as an additional training signal in the training of an object counting model, a confluence determination model, or a common model that performs an object count and a confluence determination.

14 . The method according to claim 1 ,
wherein the microscope image and the instance segmentation mask calculated therewith are used as training data for an instance segmentation model; or wherein the instance segmentation mask calculated with the microscope image and an image which is registered in relation to the microscope image and which has been captured with a different contrast method than the microscope image are used as training data for an instance segmentation model; or wherein the method further comprises: generating bounding boxes from the instance segmentation mask and using the bounding boxes and the microscope image or an image which is registered in relation to the microscope image and which has been captured with a different contrast method than the microscope image as training data for a detection model.

15 . The method according to claim 14 ,
wherein a new instance segmentation mask is calculated from the microscope image with the instance segmentation model, wherein new positions of object centers or new image areas covered by objects are determined from the new instance segmentation mask which in turn serve as the basis for the calculation of new Voronoi regions, and wherein an updated instance segmentation mask is calculated with the new Voronoi regions.

16 . The method according to claim 1 ,
wherein the microscope image is evaluated with respect to a suitability of the objects for determining an instance segmentation mask via Voronoi regions, wherein the calculation of Voronoi regions and the determination of an instance segmentation mask only occur in the event of a suitability of the objects.

17 . A non-transitory computer-readable medium storing a computer program comprising commands which, when the program is executed by a computer, cause the execution of the method according to claim 1 .

18 . A microscopy system including
a microscope for image capture; and a computing device configured to calculate an inst

Assignees

• Zeiss Carl Microscopy Gmbh

Inventors

• Amthor Manuel
• Haase Daniel

Classifications

• G06T7/12

  Edge-based segmentation · CPC title

• G06T2207/30024

  Cell structures in vitro; Tissue sections in vitro · CPC title

• G06T2207/10056

  Microscopic image · CPC title

• G06V10/82

  using neural networks · CPC title

• G06V20/69

  Microscopic objects, e.g. biological cells or cellular parts · CPC title