Deep image-to-image network learning for medical image analysis

Patent metadata

Field	Value
Publication number	US-9760807-B2
Application number	US-201615382414-A
Country	US
Kind code	B2
Filing date	Dec 16, 2016
Priority date	Jan 8, 2016
Publication date	Sep 12, 2017
Grant date	Sep 12, 2017

Abstract

A method and apparatus for automatically performing medical image analysis tasks using deep image-to-image network (DI2IN) learning. An input medical image of a patient is received. An output image that provides a result of a target medical image analysis task on the input medical image is automatically generated using a trained deep image-to-image network (DI2IN). The trained DI2IN uses a conditional random field (CRF) energy function to estimate the output image based on the input medical image and uses a trained deep learning network to model unary and pairwise terms of the CRF energy function. The DI2IN may be trained on an image with multiple resolutions. The input image may be split into multiple parts and a separate DI2IN may be trained for each part. Furthermore, the multi-scale and multi-part schemes can be combined to train a multi-scale multi-part DI2IN.

First claim

The invention claimed is:

1. A method for automatically performing a medical image analysis task on a medical image of a patient, comprising:
receiving an input medical image of a patient; and
automatically generating an output image that provides a result of a target medical image analysis task on the input medical image using a trained deep image-to-image network (DI2IN), wherein the DI2IN uses a conditional random field (CRF) energy function to estimate the output image based on the input medical image and uses a trained deep learning network to model unary and pairwise terms of the CRF energy function, wherein automatically generating the output image that provides the result of the target medical image analysis task on the input medical image using the DI2IN comprises:
generating an image pyramid with a plurality of reduced resolution images of the input medical image at an original resolution;
generating a respective output image that provides a result of the target medical image analysis task on each of the reduced resolution images of the input medical image using a sequence of trained DI2INs;
dividing the input medical image at the original resolution into a plurality of parts;
generating a respective output image that provides a result of the target medical image analysis task on each of the plurality of parts of the original resolution input medical image using a respective trained DI2IN for each of the plurality of parts; and
aggregating the output images that provide the results of the target medical image analysis task on each of the plurality of parts to generate a final output image that provides the result of the target medical image analysis task on the input medical image.

2. The method of claim 1 , wherein the target medical image analysis task is detection of one or more anatomical landmarks in the input medical image, and the estimated output image is one of a mask image in which only locations of the one or more anatomical landmarks have non-zero pixel or voxel values or an image with a Gaussian-like circle defined surrounding locations of the one or more anatomical landmarks.

3. The method of claim 1 , wherein the target medical image analysis task is detection of an anatomy of interest in the input medical image, and the estimated output image is a mask image in which only pixels or voxels located within a bounding box of the anatomy of interest have non-zero values.

4. The method of claim 1 , wherein the target medical image analysis task is segmentation of one or more anatomies of interest in the input medical image, and the estimated output image is one of a mask image in which only pixels or voxels located within boundaries of the one or more anatomies of interest have non-zero values or an image with a Gaussian-like band defined surrounding boundaries of the one or more anatomies of interest.

5. The method of claim 1 , wherein the target medical image analysis task is lesion detection, segmentation, and characterization, and the estimated output image is a multi-label mask image in which only pixels or voxels within lesion boundaries of one or more lesions have non-zero values assigned to each of the one or more lesions corresponding to a lesion type for each lesion.

6. The method of claim 1 , wherein the target medical image analysis task is an image denoising task, and the estimated output image is a reduced noise image of the input medical image.

7. The method of claim 1 , wherein the input medical image is a medical image in a source domain, the target medical image analysis task is cross-domain image synthesis, and the estimated output image is a synthesized medical image in a target domain corresponding the input medical image.

8. The method of claim 1 , wherein receiving the input medical image includes receiving the input medical image in a pair of input medical images acquired using different imaging modalities, the target medical image analysis task is registration of the pair of input medical images, and the estimated output image is a deformation field that provides the registration between the pair of input medical images.

9. The method of claim 1 , wherein receiving the input medical image includes receiving the input medical image in a set of input medical images, the target medical image analysis task is a quantitative parametric mapping task, and the estimated output image is a set of quantitative parameters that generate the set of input medical images given a pointwise generative model.

10. The method of claim 1 , further comprising, in a training stage prior to receiving the input medical image of the patient:
defining a type of output image that provides a result of the target medical image analysis task;
receiving a plurality of input training images;
receiving or generating corresponding output training images for the plurality of input training images, resulting in a training set of paired input and output training images; and
training the DI2IN that uses the CRF energy function by learning weight parameters of the deep learning network that models the unary and pairwise terms of the CRF energy function that result in a maximum likelihood for paired input and output training images over the training set of paired input and output training images.

11. The method of claim 1 , wherein automatically generating the output image that provides the result of the target medical image analysis task on the input medical image using the trained deep image-to-image network (DI2IN) comprises:
maximizing a likelihood of the CRF energy function given the input medical image and a set of learned weight parameters of the trained deep learning network, wherein the trained deep learning network calculates the unary and pairwise terms of the CRF energy function based on the input medical image, the estimated output image, and the set of learned weight parameters of the trained deep learning network.

12. The method of claim 1 , wherein the respective output image generated for each of the plurality of reduced resolution images of the input medical image defines a region of interest that is cropped from an image at a higher resolution.

13. An apparatus for automatically performing a medical image analysis task on a medical image of a patient, comprising:
means for receiving an input medical image of a patient; and
means for automatically generating an output image that provides a result of a target medical image analysis task on the input medical image using a trained deep image-to-image network (DI2IN), wherein the DI2IN uses a conditional random field (CRF) energy function to estimate the output image based on the input medical image and uses a trained deep learning network to model unary and pairwise terms of the CRF energy function, wherein the means for automatically generating the output image that provides the result of the target medical image analysis task on the input medical image using the DI2IN comprises:
means for generating an image pyramid with a plurality of reduced resolution images of the input medical image at an original resolution;
means for generating a respective output image that provides a result of the target medical image analysis task on each of the reduced resolution images of the input medical image using a sequence of trained DI2INs;
means for dividing the input medical image at the original resolution into a plurality of parts;
means for generating a respective output image that provides a result of the target medical image analysis task on each of the plurality of parts of the original resolution input medical image using a respective trained DI2IN for each of the plurality of parts; and

Assignees

• Siemens Healthcare Gmbh

Inventors

• Zhou S Kevin
• Comaniciu Dorin
• Georgescu Bogdan
• Zheng Yefeng
• Liu David
• Xu Daguang

Classifications

• G06T7/0014 Primary

  using an image reference approach · CPC title

• G06V30/194

  References adjustable by an adaptive method, e.g. learning · CPC title

• G06F18/295

  Markov models or related models, e.g. semi-Markov models; Markov random fields; Networks embedding Markov models · CPC title

• G06V10/454

  Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN] · CPC title

• G06T7/174

  involving the use of two or more images · CPC title