User interface configured to facilitate user annotation for instance segmentation within biological samples

Patent metadata

Field	Value
Publication number	US-11145058-B2
Application number	US-202016846181-A
Country	US
Kind code	B2
Filing date	Apr 10, 2020
Priority date	Apr 11, 2019
Publication date	Oct 12, 2021
Grant date	Oct 12, 2021

Abstract

Novel tools and techniques are provided for implementing digital microscopy imaging using deep learning-based segmentation via multiple regression layers, implementing instance segmentation based on partial annotations, and/or implementing user interface configured to facilitate user annotation for instance segmentation. In various embodiments, a computing system might generate a user interface configured to collect training data for predicting instance segmentation within biological samples, and might display, within a display portion of the user interface, the first image comprising a field of view of a biological sample. The computing system might receive, from a user via the user interface, first user input indicating a centroid for each of a first plurality of objects of interest and second user input indicating a border around each of the first plurality of objects of interest. The computing system might train an AI system to predict instance segmentation of objects of interest in images of biological samples.

First claim

What is claimed is:

1. A method, comprising:
generating, with a computing system, a user interface configured to collect training data using at least one of full annotation or partial annotation of objects of interest within images of biological samples;
displaying, with the computing system and within a display portion of the user interface, a first image comprising a field of view (“FOV”) of a first biological sample;
receiving, with the computing system and from a user via the user interface, a first user input that indicates a presence or location of each of a first plurality of objects of interest contained within the first image displayed in the display portion of the user interface;
generating, with the computing system, a border around each of the first plurality of objects of interest, based at least in part on a location for each of the first plurality of objects within the first image identified by the first user input and based at least in part on analysis of pixels in or around the corresponding location using an algorithm; and
generating, with the computing system, at least one of a second image or an annotation dataset based on the first image, the second image comprising data regarding location of each of the first plurality of objects of interest within the first image based on the received first user input and the generated border around each of the first plurality of objects of interest identified by the received first user input, the annotation dataset comprising at least one of pixel location data or coordinate data for each of the first plurality of objects within the first image based on the first user input and the generated border around each of the first plurality of objects of interest identified by the received first user input.

2. The method of claim 1 , wherein the computing system comprises one of a computing system disposed in a work environment, a remote computing system disposed external to the work environment and accessible over a network, a web server, a web browser, or a cloud computing system, wherein the work environment comprises at least one of a laboratory, a clinic, a medical facility, a research facility, a healthcare facility, or a room.

3. The method of claim 1 , wherein the first biological sample comprises one of a human tissue sample, an animal tissue sample, or a plant tissue sample, wherein the objects of interest comprise at least one of normal cells, abnormal cells, damaged cells, cancer cells, tumors, subcellular structures, or organ structures.

4. The method of claim 1 , further comprising:
receiving, with the computing system and from the user via the user interface, a second user input that indicates movement of a point within one of the first plurality of objects of interest from a previous position to a new position within the first image; and
generating, with the computing system, a new border around the one of the first plurality of objects of interest contained within the first image displayed in the display portion of the user interface, based at least in part on the new position of the point within the one of the first plurality of objects of interest within the first image denoted by the second user input and based at least in part on analysis of pixels in or around the new position of the point within the one of the first plurality of objects of interest using the algorithm, the new border replacing the previously generated border around the one of the first plurality of objects of interest.

5. The method of claim 4 , further comprising:
receiving, with the computing system and from the user via the user interface, a third user input that indicates partial annotation of one of a second plurality of objects of interest contained within the first image displayed in the display portion of the user interface; and
generating, with the computing system, a partial annotation symbol in the first image identifying a location of a centroid without a border for the one of the second plurality of objects of interest, based at least in part on a position of the third user input within the first image.

6. The method of claim 5 , further comprising:
receiving, with the computing system and from the user via the user interface, a fourth user input that indicates either that one of the third plurality of objects of interest is unknown or that an instance class of one of the third plurality of objects of interest should be switched to another instance class; and
generating, with the computing system, an unknown annotation symbol in the first image identifying a location of an unknown object denoted by the fourth user input, based at least in part on a position of the fourth user input within the first image, or switching, with the computing system, an instance class of a selected one of the third plurality of objects of interest to another instance class selected by the fourth user input.

7. The method of claim 6 , wherein the first user input comprises one of a click input or a bounding region input, wherein the click input defines a location of a centroid of one first object among the first plurality of objects of interest identified by the click input, wherein the bounding region input defines an area within the first image that marks an outer limit of a border of one second object among the first plurality of objects of interest identified by the bounding region input, wherein the bounding region input comprises one of a rectangular bounding region input, a circular bounding region input, a polygon placement input, or a line placement input, wherein the second user input comprises a click and drag input, wherein the third user input comprises a double click input, wherein the third user input comprises one of selection or deselection of a border around the one of the second plurality of objects of interest, wherein the fourth user input comprises one of a shift plus mouse click input or a key plus mouse click input, wherein the fourth user input comprises one of a toggling between full annotation and unknown annotation or a switch between instance classes from a list of instance classes.

8. The method of claim 1 , further comprising:
training an artificial intelligence (“AI”) system to generate or update an AI model to predict instances of objects of interest in the first biological sample based at least in part on a plurality of sets of at least two images that are generated based on the at least one of the second image or the annotation dataset, each of the at least two images among the plurality of sets of at least two images being different from each other, wherein training the AI system to generate or update the AI model to predict instances of objects of interest based at least in part on the plurality of sets of at least two images comprises:
encoding, with the computing system and using an encoder, the at least one of the second image or the annotation dataset to generate a third encoded image and a fourth encoded image, the fourth encoded image being different from the third encoded image;
training the AI system to generate or update the AI model to predict instances of objects of interest based at least in part on the third encoded image and the fourth encoded image;
generating, using the AI model that is generated or updated by the AI system, a fifth image and a sixth image based on the first image, the sixth image being different from the fifth image; and
decoding, with the computing system and using a decoder, the fifth image and the sixth image to generate a seventh image, the seventh image comprising predicted labeling of instances of objects of interest in the first biological sample.

9. The method of claim 8 , wherein the AI system comprises at least one of a mach

Assignees

• Agilent Technologies Inc

Inventors

• Arbel Elad
• Remer Itay
• Ben-Dor Amir

Classifications

• G06V20/698

  Matching; Classification · CPC title

• G06V20/695

  Preprocessing, e.g. image segmentation · CPC title

• G06V10/945

  User interactive design; Environments; Toolboxes · CPC title

• G06V10/809

  of classification results, e.g. where the classifiers operate on the same input data · CPC title

• G06V10/7753

  Incorporation of unlabelled data, e.g. multiple instance learning [MIL] · CPC title