Multi-modal image registration via modality-neutral machine learning transformation

Patent metadata

Field	Value
Publication number	US-2023260142-A1
Application number	US-202217648696-A
Country	US
Kind code	A1
Filing date	Jan 24, 2022
Priority date	Jan 24, 2022
Publication date	Aug 17, 2023
Grant date	—

Abstract

Systems/techniques that facilitate multi-modal image registration via modality-neutral machine learning transformation are provided. In various embodiments, a system can access a first image and a second image, where the first image can depict an anatomical structure according to a first imaging modality, and where the second image can depict the anatomical structure according to a second imaging modality that is different from the first imaging modality. In various aspects, the system can generate, via execution of a machine learning model on the first image and the second image, a modality-neutral version of the first image and a modality-neutral version of the second image. In various instances, the system can register the first image with the second image, based on the modality-neutral version of the first image and the modality-neutral version of the second image.

First claim

What is claimed is:

1 . A system, comprising:
a processor that executes computer-executable components stored in a computer-readable memory, the computer-executable components comprising:
a receiver component that accesses a first image and a second image, wherein the first image depicts an anatomical structure according to a first imaging modality, and wherein the second image depicts the anatomical structure according to a second imaging modality that is different from the first imaging modality;
a modality-neutral component that generates, via execution of a machine learning model on the first image and the second image, a modality-neutral version of the first image and a modality-neutral version of the second image; and
an execution component that registers the first image with the second image, based on the modality-neutral version of the first image and the modality-neutral version of the second image.

2 . The system of claim 1 , wherein an intensity distribution of the first image differs by more than a threshold amount from an intensity distribution of the second image, and wherein an intensity distribution of the modality-neutral version of the first image differs by less than the threshold amount from an intensity distribution of the modality-neutral version of the second image.

3 . The system of claim 1 , wherein the machine learning model is a deep learning neural network, wherein the modality-neutral component feeds both the first image and the second image as input to the deep learning neural network, and wherein the deep learning neural network produces as output both the modality-neutral version of the first image and the modality-neutral version of the second image.

4 . The system of claim 1 , wherein the computer-executable components further comprise:
a field component that generates a registration field based on the modality-neutral version of the first image and the modality-neutral version of the second image, wherein the registration field is a vector field indicating how pixels or voxels of the modality-neutral version of the first image should be moved to facilitate registration with the modality-neutral version of the second image, and wherein the execution component registers the first image with the second image by transforming the first image according to the registration field.

5 . The system of claim 4 , wherein the field component feeds both the modality-neutral version of the first image and the modality-neutral version of the second image as input to a deep learning registration model, and wherein the deep learning registration model produces as output the registration field.

6 . The system of claim 1 , wherein the computer-executable components further comprise:
a training component that trains the machine learning model to generate modality-neutral versions of inputted image pairs.

7 . The system of claim 6 , wherein the training component trains the machine learning model via a loss function that includes:
a first error term that is based on a pixel-to-pixel or voxel-to-voxel difference between the modality-neutral version of the first image and the modality-neutral version of the second image; a second error term that is based on an intensity-distribution-to-intensity-distribution difference between the modality-neutral version of the first image and the modality-neutral version of the second image; and a third error term that is based a first difference between the first image and the modality-neutral version of the first image, and that is based on a second difference between the second image and the modality-neutral version of the second image.

8 . The system of claim 1 , wherein the first imaging modality is one of a computed tomography imaging modality, a magnetic resonance imaging modality, an ultrasound imaging modality, an X-ray imaging modality, or a positron emission tomography imaging modality, and wherein the second imaging modality is a different one of the computed tomography imaging modality, the magnetic resonance imaging modality, the ultrasound imaging modality, the X-ray imaging modality, or the positron emission tomography imaging modality.

9 . A computer-implemented method, comprising:
accessing, by a device operatively coupled to a processor, a first image and a second image, wherein the first image depicts an anatomical structure according to a first imaging modality, and wherein the second image depicts the anatomical structure according to a second imaging modality that is different from the first imaging modality; generating, by the device and via execution of a machine learning model on the first image and the second image, a modality-neutral version of the first image and a modality-neutral version of the second image; and registering, by the device, the first image with the second image, based on the modality-neutral version of the first image and the modality-neutral version of the second image.

10 . The computer-implemented method of claim 9 , wherein an intensity distribution of the first image differs by more than a threshold amount from an intensity distribution of the second image, and wherein an intensity distribution of the modality-neutral version of the first image differs by less than the threshold amount from an intensity distribution of the modality-neutral version of the second image.

11 . The computer-implemented method of claim 9 , wherein the machine learning model is a deep learning neural network, wherein the deep learning neural network receives as input both the first image and the second image, and wherein the deep learning neural network produces as output both the modality-neutral version of the first image and the modality-neutral version of the second image.

12 . The computer-implemented method of claim 9 , further comprising:
generating, by the device, a registration field based on the modality-neutral version of the first image and the modality-neutral version of the second image, wherein the registration field is a vector field indicating how pixels or voxels of the modality-neutral version of the first image should be moved to facilitate registration with the modality-neutral version of the second image; and wherein the registering the first image with the second image includes transforming, by the device, the first image according to the registration field.

13 . The computer-implemented method of claim 12 , wherein the generating the registration field includes feeding, by the device, both the modality-neutral version of the first image and the modality-neutral version of the second image as input to a deep learning registration model, and wherein the deep learning registration model produces as output the registration field.

14 . The computer-implemented method of claim 9 , further comprising:
training, by the device, the machine learning model to generate modality-neutral versions of inputted image pairs.

15 . The computer-implemented method of claim 14 , wherein the training the machine learning model is facilitated by a loss function that includes:
a first error term that is based on a pixel-to-pixel or voxel-to-voxel difference between the modality-neutral version of the first image and the modality-neutral version of the second image; a second error term that is based on an intensity-distribution-to-intensity-distribution difference between the modality-neutral version of the first image and the modality-neutral version of the second image; and a third error term that is based a first difference between the first image and the modality-neutral version of the first image, an

Assignees

• Ge Prec Healthcare Llc

Inventors

• Chatterjee Sudhanya
• Shanbhag Dattesh Dayanand
• Shriram Krishna Seetharam

Classifications

• G06T7/33 Primary

  using feature-based methods · CPC title

• G06T2207/20081

  Training; Learning · CPC title

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title

• G06T2207/10072

  Tomographic images · CPC title

• G06T7/30 Primary

  Determination of transform parameters for the alignment of images, i.e. image registration · CPC title