Joint segmentation and characteristics estimation in medical images

Patent metadata

Field	Value
Publication number	US-10229493-B2
Application number	US-201615234426-A
Country	US
Kind code	B2
Filing date	Aug 11, 2016
Priority date	Mar 16, 2016
Publication date	Mar 12, 2019
Grant date	Mar 12, 2019

Abstract

Jointly determining image segmentation and characterization. A computer-generated image of an organ may be received. Organ characteristics estimation may be performed to predict the organ characteristics considering organ segmentation. Organ segmentation may be performed to delineate the organ in the image considering the organ characteristics. A feedback loop feeds the organ characteristics estimation to determine the organ segmentation, and feeds back the organ segmentation to determine the organ characteristics estimation.

First claim

We claim:

1. A computer-implemented method of jointly determining image segmentation and characterization, the method performed by one or more processors, the method comprising:
training an organ characteristics computer model that predicts a characteristic of an organ from a given image of the organ and a shape boundary of the organ in the given image, the organ characteristics computer model trained based on ground truth training images, shapes and organ characteristics;
training a segmentation computer model comprising an ensemble of shape regressors that predicts an organ segmentation, given training images, initial seed segmentations and the trained organ characteristics computer model, the training the segmentation computer model comprising:
extracting image features relative to a previous segmentation;
executing the organ characteristics computer model that predicts the characteristic of the organ;
augmenting the image features with the characteristic of the organ predicted by the organ characteristics computer model,
a shape regressor trained to map the augmented image features to a shape difference vector;
adding the trained shape regressor to the ensemble; and
repeating the extracting, executing, the augmenting and the adding in training the segmentation computer model until a training error is less than a preselected threshold value.

2. The method of claim 1 , further comprising:
receiving a test image and an initial segmentation data;
estimating test image organ characteristics by executing the organ characteristics computer model with the test image and the initial segmentation data as input;
estimating test image organ segmentation by executing the segmentation computer model with the test image, the initial segmentation data, and the test image organ characteristics.

3. The method of claim 1 , wherein each regressor in the ensemble predicts a shape increment vector.

4. The method of claim 1 , further comprising:
augmenting a training image with n number of random segmentations by randomly sampling from a shape distribution constructed using a point distribution model using the shapes associated with the training image.

5. The method of claim 1 , wherein the extracted image features comprise a difference between two pixels in the image, feature locations associated with the image features indexed with respect to currently estimated shape.

6. The method of claim 2 , wherein the organ comprises an eye and the organ characteristic comprises a cup-to-disc-ratio.

7. The method of claim 2 , further comprising:
displaying the test image organ segmentation on a display device.

8. A non-transitory computer readable storage medium storing a program of instructions executable by a machine to perform a method of jointly determining image segmentation and characterization, the method comprising:
training an organ characteristics computer model that predicts a characteristic of an organ from a given image of the organ and a shape boundary of the organ in the given image, the organ characteristics computer model trained based on ground truth training images, shapes and organ characteristics;
training a segmentation computer model comprising an ensemble of shape regressors that predicts an organ segmentation, given training images, initial seed segmentations and the trained organ characteristics computer model, the training the segmentation computer model comprising:
extracting image features relative to a previous segmentation;
executing the organ characteristics computer model that predicts the characteristic of the organ;
augmenting the image features with the characteristic of the organ predicted by the organ characteristics computer model,
a shape regressor trained to map the augmented image features to a shape difference vector;
adding the trained shape regressor to the ensemble; and
repeating the extracting, executing, the augmenting and the adding in training the segmentation computer model until a training error is less than a preselected threshold value.

9. The non-transitory computer readable storage medium of claim 8 , further comprising:
receiving a test image and an initial segmentation data;
estimating test image organ characteristics by executing the organ characteristics computer model with the test image and the initial segmentation data as input;
estimating test image organ segmentation by executing the segmentation computer model with the test image, the initial segmentation data, and the test image organ characteristics.

10. The non-transitory computer readable storage medium of claim 8 , wherein each regressor in the ensemble predicts a shape increment vector.

11. The non-transitory computer readable storage medium of claim 8 , further comprising:
augmenting a training image with n number of random segmentations by randomly sampling from a shape distribution constructed using a point distribution model using the shapes associated with the training image.

12. The non-transitory computer readable storage medium of claim 8 , wherein the extracted image features comprise a difference between two pixels in the image, feature locations associated with the image features indexed with respect to currently estimated shape.

13. The non-transitory computer readable storage medium of claim 9 , wherein the organ comprises an eye and the organ characteristic comprises a cup-to-disc-ratio.

14. The non-transitory computer readable storage medium of claim 9 , further comprising:
displaying the test image organ segmentation on a display device.

15. An image processing system that jointly determines image segmentation and characterization, comprising:
at least one hardware processor operable to train an organ characteristics computer model that predicts a characteristic of an organ from a given image of the organ and a shape boundary of the organ in the given image, the organ characteristics computer model trained based on ground truth training images, shapes and organ characteristics;
the at least one hardware processor further operable to train a segmentation computer model comprising an ensemble of shape regressors that predicts an organ segmentation, given training images, initial seed segmentations and the trained organ characteristics computer model, the at least one hardware processor training the segmentation computer model by at least:
extracting image features relative to a previous segmentation;
executing the organ characteristics computer model that predicts the characteristic of the organ;
augmenting the image features with the characteristic of the organ predicted by the organ characteristics computer model,
a shape regressor trained to map the augmented image features to a shape difference vector;
adding the trained shape regressor to the ensemble; and
repeating the extracting, executing, the augmenting and the adding in training the segmentation computer model until a training error is less than a preselected threshold value.

16. The system of claim 15 , wherein the at least one hardware processor is further operable to receive a test image and an initial segmentation data, estimate test image organ characteristics by executing the organ characteristics computer model with the test image and the initial segmentation data as input, and estimate test image organ segmentation by executing the segmentation computer model with the test image, the initial segmentation data, and the test image organ characteristics.

17. The system of claim 15 , wherein each regressor in the ensemble predicts a shape increment vector.

Assignees

• IBM

Inventors

• Garnavi Rahil
• Mahapatra Dwarikanath
• Roy Pallab K
• Sedai Suman

Classifications

• G06V10/26

  Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion · CPC title

• G06T7/187

  involving region growing; involving region merging; involving connected component labelling · CPC title

• G06T2207/20081

  Training; Learning · CPC title

• G06T2207/30041

  Eye; Retina; Ophthalmic · CPC title

• G06T7/11

  Region-based segmentation · CPC title