Method and system for model-based fusion of multi-modal volumetric images

Patent metadata

Field	Value
Publication number	US-9824302-B2
Application number	US-201213412869-A
Country	US
Kind code	B2
Filing date	Mar 6, 2012
Priority date	Mar 9, 2011
Publication date	Nov 21, 2017
Grant date	Nov 21, 2017

Abstract

A method and system for fusion of multi-modal volumetric images is disclosed. A first image acquired using a first imaging modality is received. A second image acquired using a second imaging modality is received. A model and of a target anatomical structure and a transformation are jointly estimated from the first and second images. The model represents a model of the target anatomical structure in the first image and the transformation projects a model of the target anatomical structure in the second image to the model in the first image. The first and second images can be fused based on estimated transformation.

First claim

The invention claimed is:

1. A method for fusion of multi-modal images, comprising:
receiving a first image acquired using a first imaging modality;
receiving a second image acquired using a second imaging modality; and
jointly estimating a model of a target anatomical structure and a transformation between the first and second images from the first and second images, wherein both the model of the target anatomical structure and the transformation between the first and second images are estimated together as a function of both of the first and second images.

2. The method of claim 1 , wherein the model of the target anatomical structure comprises a first model of the target anatomical structure in the first image, and the transformation comprises a transformation to project a second model of the target anatomical structure in the second image to the first model of the target anatomical structure.

3. The method of claim 2 , wherein the transformation includes an affine transformation and a nonlinear warping transformation.

4. The method of claim 1 , further comprising:
generating a fused image from the first and second images using the transformation.

5. The method of claim 1 , wherein the step of jointly estimating a model of a target anatomical structure and a transformation between the first and second images from the first and second images comprises:
detecting landmarks independently in each of the first and second images;
estimating an initial transformation using the landmarks detected in the first and second images;
initializing the model in the first image based on the landmarks detected in the first image;
initializing a projected model in the second image using the initial transformation; and
updating the model and transformation based at least on a joint probability detected for corresponding points of the model and the projected model.

6. The method of claim 5 , wherein the step of updating the model and transformation based at least on a joint probability detected for corresponding points of the model and the projected model comprises:
updating points of the model and the projected model based on respective single modality probabilities calculated for each point on the model and the projected model and the joint probability calculated for corresponding points on the model and the projected model; and
regularizing the model and transformation based on a learned statistical shape model of the target anatomic structure.

7. The method of claim 6 , wherein the step of updating points of the model and the projected model based on respective single modality probabilities calculated for each point on the model and the projected model and the joint probability calculated for corresponding points on the model and the projected model comprises, for each point of the model:
sampling a plurality of candidates in the first image along a line normal to the point on the model and a plurality of candidates in the second image along a line normal to the corresponding point on the projected model;
calculating a single modality probability for each of the plurality of candidates in the first image and each of the plurality of candidates in the second image using respective trained detectors;
calculating a joint probability for each possible pair of candidates in the first and second images using a trained boosting classifier; and
calculating a total probability for each possible pair of candidates in the first and second images by multiplying the single modality probability for the candidate in the first image, the single modality probability for the candidate in the second image, and the joint probability of pair of candidates; and
selecting the pair of candidates in the first and second images having the highest total probability.

8. The method of claim 6 , wherein the transformation includes an affine transformation and a nonlinear warping transformation, and the step of regularizing the model and transformation based on a learned statistical shape model of the target anatomic structure comprises:
calculating a posterior probability of the model and transformation based on the learned statistical shape model;
updating the model to improve the posterior probability by projecting the model to a principal component analysis (PCA) shape space of the learned statistical shape model;
updating the affine transformation and the nonlinear warping transformation based on the updated model in the first image and the projected model in the second image.

9. The method of claim 6 , wherein the step of updating the model and transformation based at least on a joint probability detected for corresponding points of the model and the projected model further comprises:
repeating the steps of updating points of the model and the projected model and regularizing the model and transformation until the model and the transformation converge.

10. The method of claim 1 , wherein the first image is a pre-operative image and the second image is an intra-operative image.

11. The method of claim 10 , wherein the pre-operative image is a computed tomography (CT) image, and the intra-operative image is one of a C-arm CT image and a transesophageal echocardiogram (TEE) image.

12. The method of claim 1 , wherein the target anatomic structure is an aortic valve.

13. An apparatus for fusion of multi-modal images, comprising:
means for receiving a first image acquired using a first imaging modality;
means for receiving a second image acquired using a second imaging modality; and
means for jointly estimating a model of a target anatomical structure and a transformation between the first and second images from the first and second images, wherein both the model of the target anatomical structure and the transformation between the first and second images are estimated together as a function of both of the first and second images.

14. The apparatus of claim 13 , wherein the model of the target anatomical structure comprises a first model of the target anatomical structure in the first image, and the transformation comprises a transformation to project a second model of the target anatomical structure in the second image to the first model of the target anatomical structure.

15. The apparatus of claim 14 , wherein the transformation includes an affine transformation and a nonlinear warping transformation.

16. The apparatus of claim 13 , further comprising:
means for generating a fused image from the first and second images using the transformation.

17. The apparatus of claim 13 , wherein the means for jointly estimating a model of a target anatomical structure and a transformation between the first and second images from the first and second images comprises:
means for detecting landmarks independently in each of the first and second images;
means for estimating an initial transformation using the landmarks detected in the first and second images;
means for initializing the model in the first image based on the landmarks detected in the first image;
means for initializing a projected model in the second image using the initial transformation; and
means for updating the model and transformation based at least on a joint probability detected for corresponding points of the model and the projected model.

18. The apparatus of claim 17 , wherein the means for updating the model and transformation based at least on a joint probability detected for corresponding points of the model and the projected model comprises:
means for updating points of the model and the project

Assignees

• Grbic Sasa
• Ionasec Razvan Ioan
• Wang Yang
• Georgescu Bogdan
• Mansi Tommaso
• Comaniciu Dorin
• Zheng Yefeng
• Zhou Shaohua Kevin
• John Matthias
• Boese Jan
• Siemens Healthcare Gmbh

Inventors

• Grbic Sasa
• Ionasec Razvan Ioan
• Wang Yang
• Georgescu Bogdan
• Mansi Tommaso
• Comaniciu Dorin
• Zheng Yefeng
• Zhou Shaohua Kevin
• John Matthias
• Boese Jan

Classifications

• G06V10/803

  of input or preprocessed data · CPC title

• G06F18/251

  of input or preprocessed data · CPC title

• G06V10/757 Primary

  Matching configurations of points or features · CPC title

• G06K9/6211

  Physics · mapped topic

• G06K9/6289 Primary

  Physics · mapped topic