System and method for automated transform by manifold approximation

Patent metadata

Field	Value
Publication number	US-11620772-B2
Application number	US-201716326910-A
Country	US
Kind code	B2
Filing date	Sep 1, 2017
Priority date	Sep 1, 2016
Publication date	Apr 4, 2023
Grant date	Apr 4, 2023

Abstract

A system may transform sensor data from a sensor domain to an image domain using data-driven manifold learning techniques which may, for example, be implemented using neural networks. The sensor data may be generated by an image sensor, which may be part of an imaging system. Fully connected layers of a neural network in the system may be applied to the sensor data to apply an activation function to the sensor data. The activation function may be a hyperbolic tangent activation function. Convolutional layers may then be applied that convolve the output of the fully connected layers for high level feature extraction. An output layer may be applied to the output of the convolutional layers to deconvolve the output and produce image data in the image domain.

First claim

What is claimed is:

1. A medical imaging system comprising:
an image sensor configured to acquire signal data from a patient, wherein the signal data is in a signal domain;
a data-driven, manifold-learning, neural network configured to receive the signal data from the image sensor and to transform the signal data from the signal domain to an image domain to produce an image of the patient; and
a display configured to display the image of the patient.

2. The medical imaging system of claim 1 wherein the image sensor includes at least one of:
a radio frequency (RF) system of a magnetic resonance imaging (MRI) system and wherein the signal data comprises magnetic resonance data;
an x-ray detector of a computed tomography (CT) system and wherein the signal data comprises x-ray attenuation data;
a gamma ray detector of an emission tomography system and wherein the signal data comprises emission tomography data;
an ultrasound transducer of an ultrasound system and wherein the signal data comprises ultrasound data; and
an optical sensor of an optical imaging system and wherein the signal data comprises optical imaging data.

3. The system of claim 1 , wherein the data-driven, manifold-learning, neural network includes an input layer connected to a first hidden layer.

4. The system of claim 3 , wherein the first connected layer is an n 2 ×1 first connected layer and the input layer is fully connected to the n 2 ×1 first hidden layer.

5. The system of claim 4 , wherein the n 2 ×1 first hidden layer is activated by a non-linear activation function.

6. The system of claim 5 , wherein the non-linear activation function is a hyperbolic tangent function.

7. The system of claim 5 , wherein n 2 ×1 first hidden layer is fully connected to a n 2 ×1 second hidden layer, which produces a n×n matrix when applied to n 2 ×1 first hidden layer.

8. The system of claim 7 , wherein the data-driven, manifold-learning, neural network includes fully connected layers that represent affine mapping followed by an activation function.

9. The system of claim 8 , wherein the activation function is given by g(χ)=s(Wχ+b) where g(χ) is a matrix resulting from the application of the n 2 ×1 first hidden layer to the input layer χ where W is a n 2 ×n 2 weight matrix, where b is an offset vector of dimensionality n 2 , and where s is the activation function.

10. A system comprising:
an input configured to receive signal data in a signal domain from an image sensor configured to generate the signal data, wherein the signal data corresponds to a captured image; and
a processor configured to implement a data-driven, manifold-learning, neural network configured to receive the signal data from the image sensor and to supply the signal data from the image sensor to the data-driven, manifold-learning, neural network to transform the signal data from the signal domain to an image domain to produce the captured image.

11. The system of claim 10 , wherein the data-driven, manifold-learning, neural network is configured to transform the signal data from the signal domain to the image domain by:
applying, a plurality of fully connected layers of the data-driven, manifold-learning, neural network to the signal data to produce a matrix.

12. The system of claim 11 , wherein the plurality of fully connected layers comprises:
a first hidden layer configured to operate on signal data using matrix multiplication followed by an activation function; and
a second hidden layer configured to produce the matrix from the first hidden layer, wherein the matrix has dimensions corresponding to dimensions of the signal data.

13. The system of claim 12 , wherein the plurality of fully connected layers further comprises:
an input layer configured to separate real components of the signal data from imaginary components of the signal data and to concatenate the real components and the imaginary components to produce an input vector; and
wherein the first hidden layer is applied to the input vector.

14. The system of claim 11 , wherein the activation function is a hyperbolic tangent activation function.

15. The system of claim 11 , wherein the data-driven, manifold-learning, neural network is further configured to transform the signal data from the signal domain to the image domain using the neural network by:
applying, with a plurality of convolutional layers of the neural network, a predetermined number of convolutional filters to the matrix.

16. The system of claim 15 , wherein the data-driven, manifold-learning, neural network is further configured to transform the signal data from the signal domain to the image domain using the neural network by:
applying, with a deconvolutional layer of the neural network, a predetermined number of deconvolutional filters to a convolutional layer of the plurality of convolutional layers to produce image data in the image domain that corresponds to the captured image.

17. The system of claim 10 , wherein the image sensor includes at least one of:
a radio frequency (RF) system of a magnetic resonance imaging (MRI) system and wherein the signal data comprises magnetic resonance data;
an x-ray detector of a computed tomography (CT) system and wherein the signal data comprises x-ray attenuation data;
a gamma ray detector of an emission tomography system and wherein the signal data comprises emission tomography data;
an ultrasound transducer of an ultrasound system and wherein the signal data comprises ultrasound data; and
an optical sensor of an optical imaging system and wherein the signal data comprises optical imaging data.

18. A method comprising:
generating, with an image sensor, signal data in a signal domain, wherein the signal data corresponds to a captured image;
receiving, with a processor, the signal data from the image sensor; and
executing, with the processor, a data-driven, manifold-learning, neural network for transforming the signal data from a signal domain to an image domain using a neural network to produce the captured image.

19. The method of claim 18 , wherein executing the data-driven, manifold-learning, neural network for transforming the signal data from the signal domain to the image domain using the neural network comprises executing instructions for:
applying a plurality of fully connected layers of the data-driven, manifold-learning, neural network to the signal data to produce a matrix.

20. The method of claim 19 , wherein applying the plurality of fully connected layers of the data data-driven, manifold-learning, neural network to the signal data to produce the matrix comprises:
separating, at an input layer of the plurality of fully connected layers, real components of the signal data from imaginary components of the signal data;
concatenating, at the input layer, the real components and the imaginary components to produce an input vector,
applying a first hidden layer of a plurality of fully connected layers to the input vector, and
producing, with a second hidden layer of the plurality of fully connected layers, the matrix from the first hidden layer, wherein the matrix has dimensions corresponding to dimensions of the signal data.

21. The method of claim 19 , wherein applying the first hidden layer of the plurality of fully connected layers to the input vector comprises performing matrix multiplication on the input vector before applying a hyperbolic tangent activation function.

22. The method of claim 19 , wherein executing instructions fo

Assignees

• Massachusetts Gen Hospital

Inventors

• Rosen Matthew S
• Zhu Bo
• Rosen Bruce R

Classifications

• G06T12/20

  Inverse problem, i.e. transformations from projection space into object space · CPC title

• A61B5/055

  involving electronic [EMR] or nuclear [NMR] magnetic resonance, e.g. magnetic resonance imaging · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/09

  Supervised learning · CPC title

• A61B8/5261

  combining images from different diagnostic modalities, e.g. ultrasound and X-ray · CPC title