Deep similarity learning for multimodal medical images

Patent metadata

Field	Value
Publication number	US-9922272-B2
Application number	US-201514865565-A
Country	US
Kind code	B2
Filing date	Sep 25, 2015
Priority date	Sep 25, 2014
Publication date	Mar 20, 2018
Grant date	Mar 20, 2018

Abstract

The present embodiments relate to machine learning for multimodal image data. By way of introduction, the present embodiments described below include apparatuses and methods for learning a similarity metric using deep learning based techniques for multimodal medical images. A novel similarity metric for multi-modal images is provided using the corresponding states of pairs of image patches to generate a classification setting for each pair. The classification settings are used to train a deep neural network via supervised learning. A multi-modal stacked denoising auto encoder (SDAE) is used to pre-train the neural network. A continuous and smooth similarity metric is constructed based on the output of the neural network before activation in the last layer. The trained similarity metric may be used to improve the results of image fusion.

First claim

We claim:

1. A method for similarity metric learning for multimodal medical image data, the method comprising:
receiving a first set of image data of a volume, wherein the first set of image data is captured with a first imaging modality;
receiving a second set of image data of the volume, wherein the second set of image data is captured with a second imaging modality;
aligning the first set of image data and the second set of image data;
training a first set of parameters with a multimodal stacked denoising auto encoder to generate a shared feature representation of the first set of image data and the second set of image data, the multimodal stacked denoising auto encoder comprising a first layer with independent and parallel denoising auto encoders;
training a second set of parameters with a denoising auto encoder to generate a transformation of the shared feature representation;
initializing, using the first set of parameters and the second set of parameters, a neural network classifier;
training, using training data from the aligned first set of image data and the second set of image data, the neural network classifier to generate a similarity metric for the first and second imaging modalities, the similarity metric identifying which voxels from the first set of image data that correspond to the same position in the volume as voxels from the second set of image data; and
performing image fusion on the first set of image data and the second set of image data using the identified voxels.

2. The method of claim 1 wherein the first imaging modality is computed tomography and the second imaging modality is magnetic resonance.

3. The method of claim 1 wherein the aligning comprises rigidly aligning the first set of image data and the second set of image data.

4. The method of claim 1 wherein the aligning comprises sampling the first set of image data and the second set of image data to generate a plurality of positive training data sets and a plurality of negative training data sets.

5. The method of claim 1 wherein the multimodal stacked denoising auto encoder comprises:
the first layer comprising a first denoising auto encoder and a second denoising auto encoder; and
a second layer comprising a third denoising auto encoder.

6. The method of claim 5 wherein training the multimodal stacked denoising auto encoder comprises:
training the first denoising auto encoder to generate a first feature vector from the first set of image data;
training the second denoising auto encoder to generate a second feature vector from the second set of image data; and
training the third denoising auto encoder to generate the shared feature representation from the first feature vector and the second feature vector.

7. The method of claim 1 wherein the neural network classifier is a five layer deep neural network classifier.

8. The method of claim 7 wherein initializing the neural network classifier comprises:
initializing parameters in a first three layers of the neural network classifier with parameters from the multimodal stacked denoising auto encoder; and
initializing parameters in a fourth layer of the neural network classifier with parameters from the denoising auto encoder.

9. The method of claim 8 wherein initializing the neural network classifier further comprises:
initializing missing parameters in the neural network classifier with zeros.

10. A system comprising:
a first scanner configured to capture a first set of image data of a volume with a first imaging modality;
a second scanner configured to capture a second set of image data of the volume with a second imaging modality; and
a processor configured to:
receive, from the first scanner and the second scanner over a network, the first set of image data and the second set of image data;
rigidly align the first set of image data and the second set of image data;
train a first set of parameters with a multimodal stacked denoising auto encoder to generate a shared feature representation of the first set of image data and the second set of image data, the multimodal stacked denoising auto encoder comprising a first layer with independent and parallel denoising auto encoders;
train a second set of parameters with a denoising auto encoder to generate a transformation of the shared feature representation;
initialize, using the first set of parameters and the second set of parameters, a deep neural network classifier;
train, using training data from the aligned first set of image data and the second set of image data, the deep neural network classifier to generate a similarity metric for the first and second imaging modalities, the similarity metric identifying which voxels from the first set of image data that correspond to the same position in the volume as voxels from the second set of image data; and
performing image fusion on the first set of image data and the second set of image data using the identified voxels.

11. The system of claim 10 wherein the first imaging modality is computed tomography and the second imaging modality is magnetic resonance.

12. The system of claim 10 wherein the rigidly aligning comprises sampling the first set of image data and the second set of image data to generate a plurality of positive training data sets and a plurality of negative training data sets.

13. The system of claim 10 wherein the multimodal stacked denoising auto encoder comprises:
the first layer comprising a first denoising auto encoder and a second denoising auto encoder; and
a second layer comprising a third denoising auto encoder.

14. The system of claim 13 wherein training the multimodal stacked denoising auto encoder comprises:
training the first denoising auto encoder to generate a first feature vector from the first set of image data;
training the second denoising auto encoder to generate a second feature vector from the second set of image data; and
training the third denoising auto encoder to generate the shared feature representation from the first feature vector and the second feature vector.

15. The system of claim 10 wherein the deep neural network classifier is a five layer deep neural network classifier.

16. The system of claim 15 wherein initializing the deep neural network classifier comprises:
initializing parameters in a first three layers of the deep neural network classifier with parameters from the multimodal stacked denoising auto encoder; and
initializing parameters in a fourth layer of the deep neural network classifier with parameters from the denoising auto encoder.

17. The system of claim 14 wherein initializing the deep neural network classifier further comprises:
initializing missing parameters in the deep neural network classifier with zeros.

Assignees

• Siemens Healthcare Gmbh

Inventors

• Cheng Xi
• Zhang Li
• Zheng Yefeng

Classifications

• G06V10/803

  of input or preprocessed data · CPC title

• G06V10/82 Primary

  using neural networks · CPC title

• G06V10/761

  Proximity, similarity or dissimilarity measures · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G06F18/251

  of input or preprocessed data · CPC title