Method and system for vascular disease detection using recurrent neural networks

Patent metadata

Field	Value
Publication number	US-9767557-B1
Application number	US-201715429806-A
Country	US
Kind code	B1
Filing date	Feb 10, 2017
Priority date	Jun 23, 2016
Publication date	Sep 19, 2017
Grant date	Sep 19, 2017

Abstract

A method and apparatus for vascular disease detection and characterization using a recurrent neural network (RNN) is disclosed. A plurality of 2D cross-section image patches are extracted from a 3D computed tomography angiography (CTA) image, each extracted at a respective sampling point along a vessel centerline of a vessel of interest in the 3D CTA image. Vascular abnormalities in the vessel of interest are detected and characterized by classifying each of the sampling points along the vessel centerline based on the plurality of 2D cross-section image patches using a trained RNN.

First claim

The invention claimed is:

1. A method for vascular disease detection using a recurrent neural network, comprising:
extracting a plurality of 2D cross-section image patches from a 3D computed tomography angiography (CTA) image, wherein each of 2D cross-section image patches is extracted at a respective one of a plurality of sampling points along a vessel centerline of a vessel of interest in the 3D CTA image; and
detecting vascular abnormalities in the vessel of interest by classifying each of the plurality of sampling points along the vessel centerline based on the plurality of 2D cross-section image patches using a trained recurrent neural network (RNN).

2. The method of claim 1 , further comprising:
detecting the vessel centerline of the vessel of interest in the 3D CTA image.

3. The method of claim 1 , wherein detecting vascular abnormalities in the vessel of interest by classifying each of the plurality of sampling points along the vessel centerline based on the plurality of 2D cross-section image patches using a trained recurrent neural network (RNN) comprises:
encoding each of the 2D cross-section image patches into a respective feature vector using a trained convolutional neural network (CNN); and
classifying each of the plurality of sampling points along the vessel centerline based on the feature vectors corresponding to the plurality of 2D cross-section image patches using a trained bi-directional RNN.

4. The method of claim 3 , wherein the bi-directional RNN is a bi-directional long short-term memory (LSTM) network, and classifying each of the plurality of sampling points along the vessel centerline based on the feature vectors corresponding to the plurality of 2D cross-section image patches using a trained bi-directional RNN comprises:
inputting the feature vectors corresponding to the plurality of 2D cross-section image patches to a forward LSTM network and a backward LSTM network;
sequentially classifying each of the plurality of sampling points along the vessel centerline in a forward direction from a first sampling point to an n th sampling point using the forward LSTM network, wherein the forward LSTM network classifies each sampling point based on the feature vector corresponding to the 2D cross-section image extracted at that sampling point and the feature vectors corresponding to the 2D cross-section images extracted at sampling points previously classified by the forward LSTM network;
sequentially classifying each of the plurality of sampling points along the vessel centerline in a backward direction from the nth sampling point to the first sampling point using the backward LSTM network, wherein the backward LSTM network classifies each sampling point based on the feature vector corresponding to the 2D cross-section image extracted at that sampling point and the feature vectors corresponding to the 2D cross-section images extracted at sampling points previously classified by the backward LSTM network; and
combining classification results output by the forward LSTM and classification results output by the backward LSTM for each of the plurality of sampling points.

5. The method of claim 1 , wherein detecting vascular abnormalities in the vessel of interest by classifying each of the plurality of sampling points along the vessel centerline based on the plurality of 2D cross-section image patches using a trained recurrent neural network (RNN) comprises:
generating multiscale image information for each of the 2D cross-section image patches; and
classifying each of the plurality of sampling points along the vessel centerline based on the multiscale image information generated for the plurality of 2D cross-section image patches using the trained RNN.

6. The method of claim 5 , wherein generating multiscale image information for each of the 2D cross-section image patches comprises:
generating an image pyramid with multiple reduced resolution image patches for each of the 2D cross-section image patches.

7. The method of claim 6 , wherein the trained RNN is a bi-directional long short-term memory (LSTM) network, and classifying each of the plurality of sampling points along the vessel centerline based on the multiscale image information generated for the plurality of 2D cross-section image patches using the trained RNN comprises, for each of the plurality of sampling points along the vessel centerline:
encoding each of multiple reduced resolution image patches generated for the 2D cross-section image patch extracted at a current sampling point into a respective feature vector for each resolution using a respective trained convolutional neural network (CNN) for each resolution;
concatenating the feature vectors encoded from all of the multiple reduced resolution images into a concatenated feature vector;
inputting the concatenated feature vector the current sampling point based on a learned input gating function of the bi-directional LSTM; and
classifying the current sampling point based on the input concatenated feature vector for the current sampling point and a memory state of the bi-directional LSTM at the current sampling point.

8. The method of claim 6 , wherein the trained RNN is a bi-directional long short-term memory (LSTM) network, and classifying each of the plurality of sampling points along the vessel centerline based on the multiscale image information generated for the plurality of 2D cross-section image patches using the trained RNN comprises, for each of the plurality of sampling points along the vessel centerline:
encoding each of multiple reduced resolution image patches generated for the 2D cross-section image patch extracted at a current sampling point into a respective feature vector for each resolution using a respective trained convolutional neural network (CNN) for each resolution;
inputting the feature vector encoded for each resolution based on a respective learned gating function of the bi-directional LSTM for each resolution;
summing the feature vectors input by the respective learned gating functions for each resolution; and
classifying the current sampling point based on the summed input feature vectors for the current sampling point and a memory state of the bi-directional LSTM at the current sampling point.

9. The method of claim 6 , wherein classifying each of the plurality of sampling points along the vessel centerline based on the multiscale image information generated for the plurality of 2D cross-section image patches using the trained RNN comprises, for each of the plurality of sampling points along the vessel centerline:
encoding each of multiple reduced resolution image patches generated for the 2D cross-section image patch extracted at a current sampling point into a respective feature vector for each resolution using a respective trained convolutional neural network (CNN) for each resolution;
inputting the encoded feature vector for each resolution to a respective trained long short-term memory (LSTM) network for each resolution; and
sequentially performing classification of the current sample point by the respective trained LSTM for each resolution, starting with a lowest resolution level LSTM, wherein an output of each LSTM other than a highest resolution level LSTM is input to a subsequent LSTM at a next resolution level and the highest resolution level LSTM outputs a classification result for the current sampling point.

10. The method of claim 1 , wherein detecting vascular abnormalities in the vessel of interest by classifying each of the plurality of sampling points along the vessel centerline based on the plurality of 2D cross-section image patches using a trained recurrent neural network (RNN) comprises:
detecting vascular abnormalit

Assignees

• Siemens Healthcare Gmbh

Inventors

• Gulsun Mehmet A
• Zheng Yefeng
• Sharma Puneet
• Georgescu Bogdan
• Comaniciu Dorin

Classifications

• G06V30/19173

  Classification techniques · CPC title

• G06V10/82

  using neural networks · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G06N3/044

  Recurrent networks, e.g. Hopfield networks · CPC title

• G06F18/24133

  Distances to prototypes · CPC title