Modality-agnostic method for medical image representation

Patent metadata

Field	Value
Publication number	US-10762398-B2
Application number	US-201815986065-A
Country	US
Kind code	B2
Filing date	May 22, 2018
Priority date	Apr 30, 2018
Publication date	Sep 1, 2020
Grant date	Sep 1, 2020

Abstract

Techniques for the operation and use of a model that learns the general representation of multimodal images is disclosed. In various examples, methods from representation learning are used to find a common basis for representation of medical images. These include aspects of encoding, fusion, and downstream tasks, with use of the general representation and model. In an example, a method for generating a modality-agnostic model includes receiving imaging data, encoding the imaging data by mapping data to a latent representation, fusing the encoded data to conserve latent variables corresponding to the latent representation, and training a model using the latent representation. In an example, a method for processing imaging data using a trained modality-agnostic model includes receiving imaging data, encoding the data to the defined encoding, processing the encoded data with a trained model, and performing imaging processing operations based on output of the trained model.

First claim

What is claimed is:

1. A computer implemented method for generating a modality-agnostic image processing model, the method comprising:
receiving imaging data from multiple imaging modalities;
encoding the imaging data based on applying a defined data encoding for each modality of the multiple imaging modalities, the encoding comprising:
mapping the imaging data to respective latent representations corresponding to the multiple imaging modalities, and
producing respective latent variables from applying the defined data encoding to the imaging data, wherein the respective latent variables correspond to a spatial representation of the respective latent representations;

fusing the encoded imaging data, the fusing comprising:
mapping the respective latent representations to a fused latent representation of the encoded imaging data,
wherein the mapping uses the respective latent representations and the respective latent variables to determine the fused latent representation, and
wherein the mapping combines the respective latent representations into the fused latent representation while conserving structure indicated by the respective latent variables;

training a model for medical imaging processing, using the fused latent representation of the encoded imaging data; and
outputting the trained model, the model adapted to produce an output from subsequent medical imaging data according to the training using the fused latent representation.

2. The method of claim 1 , wherein the encoding further comprises producing confidence values corresponding to the respective latent representations, and wherein the confidence values are used as weights in mapping the respective latent representations into the fused latent representation.

3. The method of claim 1 , wherein the encoding is performed with a variational autoencoder model.

4. The method of claim 3 , wherein the variational autoencoder model comprises a neural network.

5. The method of claim 3 , wherein the variational autoencoder model is trained for encoding of imaging data, and wherein the respective latent variables used in the fusing are produced from spatial latent variables used in the variational autoencoder model.

6. The method of claim 1 , wherein respective sets of the imaging data are captured from among the multiple imaging modalities, wherein the respective sets of the imaging data captured from among the multiple imaging modalities includes images produced from among a plurality of modality types or from a plurality of modes of at least one modality type.

7. The method of claim 1 , wherein the fused latent representation is produced as a fused representation image, and wherein the model is trained based on the fused representation image.

8. The method of claim 7 , wherein the fused representation image is produced using at least one of: spectral representations, covariance tapering, low rank approximations, or Gaussian Markov Random Fields.

9. The method of claim 1 , wherein the defined data encoding is provided from among a plurality of encoders corresponding to a respective type or mode of the imaging modalities.

10. The method of claim 9 , wherein the fusing of the encoded imaging data applies respective weights corresponding to the plurality of encoders in producing the fused latent representation.

11. The method of claim 10 , wherein the respective weights corresponding to the plurality of encoders are normalized exponential confidence weights.

12. The method of claim 1 , wherein the fused latent representation includes a same number of image dimensions as included in the received imaging data.

13. The method of claim 1 , further comprising,
receiving the subsequent medical imaging data, the subsequent medical imaging data provided from another modality type or mode that is not included in the received imaging data captured from multiple imaging modalities;
operating the trained model on the subsequent medical imaging data of the another modality type or mode.

14. The method of claim 1 , wherein the output produced from operating the trained model is used for performing radiotherapy planning operations.

15. A computer-implemented method for processing imaging data using a trained modality-agnostic model, the method comprising:
receiving medical imaging data of a first imaging modality;
encoding the medical imaging data based on a defined data encoding, by mapping the medical imaging data to a latent representation;
processing the encoded medical imaging data with a trained model, wherein the trained model produces an output based on the latent representation, based on training of the model to a fused latent representation of training imaging data, wherein training of the model is performed based on a mapping of a latent representation of the training imaging data obtained from at least a second imaging modality, wherein the training of the model is performed from:
encoding of the training imaging data based on a defined data encoding for each modality of the training imaging data, the encoding performed from mapping of the training imaging data to respective latent representations corresponding to each modality, and producing respective latent variables from applying the defined data encoding to the training imaging data, wherein the respective latent variables correspond to a spatial representation of the respective latent representations; and
fusing of the encoded training imaging data, the fusing performed from mapping of the respective latent representations to a fused latent representation of the encoded training imaging data, wherein the mapping uses the respective latent representations and the respective latent variables to determine the fused latent representation, and wherein the mapping combines the respective latent representations into the fused latent representation while conserving structure indicated by the respective latent variables; and

performing an image processing operation on the medical imaging data based on the output produced by the trained model.

16. The method of claim 15 , wherein the image processing operation comprises at least one of: segmentation, denoising, synthesis, classification, regression, or reconstruction.

17. The method of claim 15 , the method further comprising:
decoding the encoded imaging data; and
performing image reconstruction based on the decoded imaging data.

18. The method of claim 17 , wherein the first imaging modality comprises a first modality type or a first modality mode, wherein the trained model is a neural network, and wherein the trained model is trained using data from at least a second modality type or a second modality mode, and wherein the first modality type or first modality mode is unused in training the model.

19. The method of claim 17 , wherein the first imaging modality and the second imaging modality are respective types of imaging procedures, comprising at least two of: Magnetic resonance imaging (MRI), computed tomography (CT), Positron Emission Tomography (PET), PET-CT, Ultrasound, X-Ray, Fluoroscopy, Single-photon emission computed tomography (SPECT), Elastography, Photoacoustic, Magnetoencephalography (MEG), or Electroencephalography (EEG) imaging procedures, or combinations of such imaging procedures.

20. The method of claim 19 , wherein the first imaging modality and the second imaging modality are respective types of MRI imaging modes, comprising at least two of: T1, T1 with contrast, T2, PD, SSFP, STIR, FLAIR, DIR, DWI, PWI, or fMRI.

21

Assignees

• Elekta Ab

Inventors

• Sjölund Jens Olof
• Adler Jonas Anders

Classifications

• G16H20/30 Primary

  relating to physical therapies or activities, e.g. physiotherapy, acupressure or exercising · CPC title

• G06V10/82

  using neural networks · CPC title

• G06V10/454

  Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN] · CPC title

• G06V10/764

  using classification, e.g. of video objects · CPC title

• G06F18/251

  of input or preprocessed data · CPC title