Method and system for anatomical object detection using marginal space deep neural networks

Patent metadata

Field	Value
Publication number	US-9668699-B2
Application number	US-201514709536-A
Country	US
Kind code	B2
Filing date	May 12, 2015
Priority date	Oct 17, 2013
Publication date	Jun 6, 2017
Grant date	Jun 6, 2017

Abstract

A method and system for anatomical object detection using marginal space deep neural networks is disclosed. The pose parameter space for an anatomical object is divided into a series of marginal search spaces with increasing dimensionality. A respective deep neural network is trained for each of the marginal search spaces, resulting in a series of trained deep neural networks. Each of the trained deep neural networks can evaluate hypotheses in a current parameter space using discriminative classification or a regression function. An anatomical object is detected in a medical image by sequentially applying the series of trained deep neural networks to the medical image.

First claim

The invention claimed is:

1. A method for anatomical object detection in a medical image of a patient, comprising:
receiving a medical image of a patient including a target anatomical object; and
detecting a pose of the target anatomical object in the medical image in a series of marginal parameter spaces of increasing dimensionality using a respective trained deep neural network for each of the marginal parameter spaces, wherein each respective trained deep neural network inputs image patches of the medical image, samples pixels or voxels in each input image patch in a sparse sampling pattern adaptively learned during training of the respective trained deep neural network, and operates directly on the pixels or voxels sampled from each input image patch to calculate one or more candidates for the target anatomical object in a respective one of the marginal parameter spaces and wherein, for each of the marginal parameter spaces, a respective cascade of trained shallow sparse neural networks filters image patches of the medical image to remove negative image patches prior to the remaining image patches being input to the respective trained deep neural network for that marginal parameter space.

2. The method of claim 1 , wherein the respective trained deep neural network for at least one of the marginal parameter spaces comprises a discriminative deep neural network that inputs hypothesis image patches in the respective marginal parameter space, and calculates, for each of the hypothesis image patch, a probability that the hypothesis image patch is an image patch of the target anatomical object in the respective marginal parameter space.

3. The method of claim 1 , wherein the respective trained deep neural network for at least one of the marginal parameter spaces comprises a deep neural network regressor that inputs hypothesis image patches in the respective marginal parameter space, and calculates, for each hypothesis image patch, a displacement vector that maps the hypothesis image patch to a predicted image patch of the target anatomical object in the respective marginal parameter space.

4. The method of claim 3 , wherein the deep neural network regressor further calculates, for each hypothesis image patch, a probability for the corresponding predicted image patch of the target anatomical object in the respective marginal parameter space.

5. The method of claim 3 , wherein the deep neural network regressor iteratively refines the predicted image patches of the target anatomical object in the respective marginal parameter space by inputting each predicted image patch back into the deep neural network regressor and calculating, for each predicted image patch, a displacement vector that maps the predicted image patch to a refined predicted image patch in the respective marginal parameter space.

6. The method of claim 1 , wherein the series of marginal parameter spaces comprises a position parameter space, a position-orientation parameter space, and a position-orientation-scale parameter space.

7. The method of claim 6 , wherein detecting a pose of the target anatomical object in the medical image in a series of marginal parameter spaces of increasing dimensionality using a respective trained deep neural network for each of the marginal parameter spaces comprises:
detecting position candidates in the medical image using a first trained deep neural network that inputs image patches centered at each of a plurality of voxels in the medical image;
generating position-orientation hypotheses based on the detected position candidates;
detecting position-orientation candidates in the medical image using a second trained deep neural network that inputs image patches corresponding to the position-orientation hypotheses;
generating position-orientation-scale hypotheses based on the detected position-orientation candidates; and
detecting the pose of the target anatomical object in the medical image using a third trained deep neural network that inputs image patches corresponding to the position-orientation-scale hypotheses.

8. The method of claim 7 , wherein generating position-orientation hypotheses based on the detected position candidates comprises:
rotating a respective image patch centered at each detected position candidate to a plurality of orientations sampled from a predetermined range of orientations for the target anatomical object.

9. The method of claim 7 , wherein generating position-orientation-scale hypotheses based on the detected position-orientation candidates comprises:
scaling a respective image patch corresponding to each detected position-orientation candidate to a plurality of scales sampled from a predetermined range of scales for the target anatomical object.

10. The method of claim 7 , wherein the second trained deep neural network is a trained deep neural network regressor, and generating position-orientation hypotheses based on the detected position candidates comprises:
generating a respective position-orientation hypothesis for each detected position candidate by using an image patch centered at each detected position candidate as the respective position-orientation hypothesis.

11. The method of claim 7 , wherein the second trained deep neural network is a trained deep neural network regressor, and generating position-orientation-scale hypotheses based on the detected position-orientation candidates comprises:
generating a respective position-orientation-scale hypothesis for each detected position-orientation candidate by using an image patch corresponding to each detected position-orientation candidate as the respective position-orientation-scale hypothesis.

12. The method of claim 1 , wherein the respective deep neural network for each of the marginal parameter spaces is a multi-layer deep neural network trained using a stacked denoising auto-encoder (DAE).

13. The method of claim 1 , wherein the respective deep neural network for each of the marginal parameter spaces is a multi-layer deep neural network trained using one of a convolutional neural network (CNN), a stacked restricted Boltzmann machine (RBM), or a stacked auto-encoder (AE).

14. The method of claim 1 , wherein the respective trained deep neural network for each of the marginal parameter spaces is trained by sequentially training each of a plurality of hidden layers to reconstruct respective inputs to each hidden layer in an unsupervised pre-training stage, and training the deep neural network to detect the target anatomical object in the respective marginal parameter space based on ground truth training image patches in the respective marginal parameter space in a supervised training stage.

15. The method of claim 14 , wherein the respective trained deep neural network for each of the marginal parameter spaces is trained in the supervised training stage by adding an output layer for a target output for the respective marginal parameter space and training the plurality of hidden layers and the output layer using back-propagation to minimize an error between detected image patches for the anatomical object and the ground truth training image patches in the respective marginal parameter space.

16. The method of claim 14 , wherein the respective trained deep neural network for each of the marginal parameter spaces is trained in the supervised training stage by treating an output of the plurality of hidden layers as high-level image features and training a discriminative classifier based on the high-level image features output by the plurality of hidden layers.

17. The method of claim 1 , wherein the

Assignees

• Siemens Healthcare Gmbh

Inventors

• Georgescu Bogdan
• Zheng Yefeng
• Nguyen Hien
• Singh Vivek Kumar
• Comaniciu Dorin
• Liu David

Classifications

• A61B5/7267 Primary

  involving training the classification device · CPC title

• G06V10/772 Primary

  Determining representative reference patterns, e.g. averaging or distorting patterns; Generating dictionaries · CPC title

• G06F18/28

  Determining representative reference patterns, e.g. by averaging or distorting; Generating dictionaries · CPC title

• G06V10/454

  Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN] · CPC title

• A61B5/055

  involving electronic [EMR] or nuclear [NMR] magnetic resonance, e.g. magnetic resonance imaging · CPC title