Saliency mapping by feature reduction and perturbation modeling in medical imaging

Patent metadata

Field	Value
Publication number	US-11263744-B2
Application number	US-201916707209-A
Country	US
Kind code	B2
Filing date	Dec 9, 2019
Priority date	Dec 9, 2019
Publication date	Mar 1, 2022
Grant date	Mar 1, 2022

Abstract

For saliency mapping, a machine-learned classifier is used to classify input data. A perturbation encoder is trained and/or applied for saliency mapping of the machine-learned classifier. The training and/or application (testing) of the perturbation encoder uses less than all feature maps of the machine-learned classifier, such as selecting different feature maps of different hidden layers in a multiscale approach. The subset used is selected based on gradients from back-projection. The training of the perturbation encoder may be unsupervised, such as using an entropy score, or semi-supervised, such as using the entropy score and a difference of a perturbation mask from a ground truth segmentation.

First claim

We claim:

1. A method for saliency mapping in a medical imaging system, the method comprising:
generating, by the medical imaging system, an image of a patient;
classifying the image with a machine-learned classifier, the classifying with the machine-learned classifier determining values for a plurality of features of the machine-learned classifier in outputting a class membership of the image;
selecting less than all features of the plurality of features of the machine-learned classifier based on the determined values to form a subset of the features;
forming a saliency map of the image from input of the determined values for only the subset of the features to a machine-learned model; and
displaying the class membership and the saliency map.

2. The method of claim 1 wherein selecting comprises performing back-propagation from the classifying by the machine-learned classifier and selecting the subset from results of the back-propagation.

3. The method of claim 2 wherein selecting comprises selecting the features having a largest gradient from the back-propagation.

4. The method of claim 3 wherein selecting comprises selecting with the largest gradient comprising a largest average gradient.

5. The method of claim 1 wherein selecting comprises separately selecting for different hidden layers of the machine-learned classifier.

6. The method of claim 1 wherein the machine-learned classifier comprises a convolutional neural network, and wherein selecting the subset comprises selecting convolutions of the convolutional neural network.

7. The method of claim 1 wherein forming comprises outputting from the machine-learned model comprising a fully convolutional network.

8. The method of claim 1 wherein forming comprises forming by the machine-learned model comprising a machine-trained perturbation encoder.

9. The method of claim 1 wherein forming comprises forming by the machine-learned model, the machine-learned model having been trained in a semi-supervised manner with a score function based on a difference between a ground truth segmentation and an estimated perturbation.

10. The method of claim 1 wherein selecting comprises selecting the subset as class sensitive features.

11. A method for machine training saliency map prediction of a machine-learned classifier, the method comprising:
classifying an input image by the machine-learned classifier;
back-propagating from a class of the classifying, the back-propagating providing gradients with respect to feature maps of the machine-learned classifier;
selecting less than all of the feature maps based on sensitivities of the feature maps to the class;
generating a perturbation mask by input of the selected features maps to a perturbation encoder;
inputting a perturbed input formed by combining the perturbation mask with the input image to the machine-learned classifier, the inputting resulting in a distorted class prediction;
computing a score from the distorted class prediction; and
altering the perturbation encoder to minimize the score.

12. The method of claim 11 wherein selecting comprises selecting separately for each of a plurality of hidden layers of the machine-learned classifier.

13. The method of claim 11 wherein the combining of the inputting comprises combining as a Hadamard product.

14. The method of claim 11 wherein selecting comprises selecting from averages of the gradients for the different feature maps.

15. The method of claim 11 further comprising storing the perturbation encoder.

16. The method of claim 11 wherein classifying comprises classifying by the machine-learned classifier comprising a convolutional neural network, the feature maps corresponding to convolution kernels, and wherein generating comprises generating by the perturbation encoder comprising a fully connected neural network.

17. The method of claim 11 wherein computing the score comprises computing from the distorted class prediction and from a difference of the perturbation mask and a ground truth segmentation.

18. A system for saliency map prediction, the system comprising:
a medical imaging scanner configured to generate data representing a patient;
an image processor configured to classify the data by application of a machine-learned model and to generate a saliency map for the classification by application of input based on gradients from back-propagation of the classification to a perturbation encoder, the perturbation encoder having been machine trained in a semi-supervised manner based on entropy and a difference of saliency from ground truth segmentations; and
a display configured to display the classification and the saliency map for the patient.

19. The system of claim 18 wherein the image processor is configured to select a sub-set of feature maps using the gradients, and wherein the input is the selected feature maps.

20. The system of claim 18 wherein the perturbation encoder was trained with a score function including the entropy and the difference.

Assignees

• Siemens Healthcare Gmbh

Inventors

• Yoo Youngjin
• Ceccaldi Pascal
• Gibson Eli
• Nadar Mariappan S

Classifications

• G06V10/462

  Salient features, e.g. scale invariant feature transforms [SIFT] · CPC title

• G06V10/82

  using neural networks · CPC title

• G06V10/764

  using classification, e.g. of video objects · CPC title

• G06F18/217

  Validation; Performance evaluation; Active pattern learning techniques · CPC title

• G06N3/045

  Combinations of networks · CPC title