Deep learning based instance segmentation via multiple regression layers

Patent metadata

Field	Value
Publication number	US-11410303-B2
Application number	US-202016846180-A
Country	US
Kind code	B2
Filing date	Apr 10, 2020
Priority date	Apr 11, 2019
Publication date	Aug 9, 2022
Grant date	Aug 9, 2022

Abstract

Novel tools and techniques are provided for implementing digital microscopy imaging using deep learning-based segmentation and/or implementing instance segmentation based on partial annotations. In various embodiments, a computing system might receive first and second images, the first image comprising a field of view of a biological sample, while the second image comprises labeling of objects of interest in the biological sample. The computing system might encode, using an encoder, the second image to generate third and fourth encoded images (different from each other) that comprise proximity scores or maps. The computing system might train an AI system to predict objects of interest based at least in part on the third and fourth encoded images. The computing system might generate (using regression) and decode (using a decoder) two or more images based on a new image of a biological sample to predict labeling of objects in the new image.

First claim

What is claimed is:

1. A method, comprising:
receiving, with a computing system, a first image, the first image comprising a field of view (“FOV”) of a first biological sample;
receiving, with the computing system, a second image, the second image comprising labeling of instances of objects of interest in the first biological sample;
encoding, with the computing system and using an encoder, the second image to generate a third encoded image, the encoding comprising:
computing, with the computing system, a centroid for each labeled instance of an object of interest in the second image; and
generating, with the computing system, the third encoded image, the third encoded image comprising highlighting of the centroid for each labeled instance of an object of interest;

encoding, with the computing system and using an encoder, the second image to generate a fourth encoded image, the encoding comprising:
computing, with the computing system, an edge or border for each labeled instance of an object of interest in the second image; and
generating, with the computing system, the fourth encoded image, the fourth encoded image comprising highlighting of the edge or border for each labeled instance of the object of interest; and

training an artificial intelligence (“AI”) system to generate or update an AI model to predict instances of objects of interest, wherein an input to the training of the AI system includes the third encoded image and the fourth encoded image.

2. The method of claim 1 , wherein the computing system comprises one of a computing system disposed in a work environment, a remote computing system disposed external to the work environment and accessible over a network, a web server, a web browser, or a cloud computing system, wherein the work environment comprises at least one of a laboratory, a clinic, a medical facility, a research facility, a healthcare facility, or a room.

3. The method of claim 1 , wherein the AI system comprises at least one of a machine learning system, a deep learning system, a neural network, a convolutional neural network (“CNN”), or a fully convolutional network (“FCN”).

4. The method of claim 1 , wherein the first biological sample comprises one of a human tissue sample, an animal tissue sample, or a plant tissue sample, wherein the objects of interest comprise at least one of normal cells, abnormal cells, damaged cells, cancer cells, tumors, subcellular structures, or organ structures.

5. The method of claim 1 , wherein training the AI system to generate or update the AI model to predict instances of objects of interest further comprises:
generating, using the AI model that is generated or updated by the AI system, a fifth image and a sixth image based on the first image, the fifth image comprising highlighting of a centroid for each predicted instance of an object of interest, the sixth image comprising highlighting of an edge or border for each predicted instance of the object of interest; and
decoding, with the computing system and using a decoder, the fifth image and the sixth image to generate a seventh image, the seventh image comprising predicted labeling of instances of objects of interest in the first biological sample.

6. The method of claim 5 , wherein training the AI system to generate or update the AI model to predict instances of objects of interest further comprises:
comparing, with the computing system, the seventh image with the second image to generate an instance segmentation evaluation result.

7. The method of claim 5 wherein:
encoding the second image to generate the third encoded image further comprises:
computing, with the computing system, first distance measures between each pixel in the third encoded image and each centroid for each labeled instance of the object of interest; and
computing, with the computing system, a first function to generate a first proximity map, the first function being a function of the first distance measures, the third encoded image comprising the first proximity map; and

encoding the second image to generate the fourth encoded image further comprises:
computing, with the computing system, second distance measures between each pixel in the fourth encoded image and a nearest edge pixel of the edge or border for each labeled instance of the object of interest; and
computing, with the computing system, a second function to generate a second proximity map, the second function being a function of the second distance measures, the fourth encoded image comprising the second proximity map.

8. The method of claim 7 , further comprising:
assigning, with the computing system, a first weighted pixel value for each pixel in the third encoded image, based at least in part on at least one of the computed first distance measures for each pixel, the first function, or the first proximity map; and
assigning, with the computing system, a second weighted pixel value for each pixel in the fourth encoded image, based at least in part on at least one of the computed second distance measures for each pixel, the second function, or the second proximity map.

9. The method of claim 7 , further comprising:
determining, with the computing system, a first pixel loss value between each pixel in the third encoded image and a corresponding pixel in the fifth image;
determining, with the computing system, a second pixel loss value between each pixel in the fourth encoded image and a corresponding pixel in the sixth image;
calculating, with the computing system, a loss value using a loss function, based on a product of the first weighted pixel value for each pixel in the third encoded image multiplied by the first pixel loss value between each pixel in the third encoded image and a corresponding pixel in the fifth image and a product of the second weighted pixel value for each pixel in the fourth encoded image multiplied by the second pixel loss value between each pixel in the fourth encoded image and a corresponding pixel in the sixth image, wherein the loss function comprises one of a mean squared error loss function, a mean squared logarithmic error loss function, a mean absolute error loss function, a Huber loss function, or a weighted sum of squared differences loss function; and
updating, with the AI system, the AI model, by updating one or more parameters of the AI model based on the calculated loss value;
wherein generating the fifth image and the sixth image comprises generating, using the updated AI model, the fifth image and the sixth image, based on the first image.

10. The method of claim 9 , wherein labeling of instances of objects of interest in the second image comprises at least one of full annotation of first instances of objects of interest that identify centroid and edge of the first instances of objects of interest, partial annotation of second instances of objects of interest that identify only centroid of the second instances of objects of interest, or unknown annotation of third instances of objects of interest that identify neither centroid nor edge.

11. The method of claim 10 , further comprising:
masking, with the computing system, the second instances of objects of interest with partial annotation in the fourth encoded image and corresponding pixels in the sixth image, without masking the second instances of objects of interest with partial annotation in the third encoded image or in the fifth image, prior to calculating the loss value; and
masking, with the computing system, the third instances of objects of interest with unknown annotation in the third encoded image and corresponding pixels in the fifth image and in the fourth encoded image and corresponding pixels in the sixth image, p

Assignees

• Agilent Technologies Inc

Inventors

• Arbel Elad
• Remer Itay
• Ben-Dor Amir

Classifications

• G06V20/698

  Matching; Classification · CPC title

• G06V20/695

  Preprocessing, e.g. image segmentation · CPC title

• G06V10/945

  User interactive design; Environments; Toolboxes · CPC title

• G06V10/809

  of classification results, e.g. where the classifiers operate on the same input data · CPC title

• G06V10/7753

  Incorporation of unlabelled data, e.g. multiple instance learning [MIL] · CPC title