Method, learning apparatus, and medical imaging apparatus for registration of images

Patent metadata

Field	Value
Publication number	US-10832392-B2
Application number	US-201816225774-A
Country	US
Kind code	B2
Filing date	Dec 19, 2018
Priority date	Dec 19, 2018
Publication date	Nov 10, 2020
Grant date	Nov 10, 2020

Abstract

A method of training a computer system for use in determining a transformation between coordinate frames of image data representing an imaged subject. The method trains a learning agent according to a machine learning algorithm, to determine a transformation between respective coordinate frames of a number of different views of an anatomical structure simulated using a 3D model. The views are images containing labels. The learning agent includes a domain classifier comprising a feature map generated by the learning agent during the training operation. The classifier is configured to generate a classification output indicating whether image data is synthesized or real images data. Training includes using unlabeled real image data to training the computer system to determine a transformation between a coordinate frame of a synthesized view of the imaged structure and a view of the structure within a real image. This is done whilst deliberately reducing the ability of the domain classifier to discriminate between a synthesized image and a real image of the structure.

First claim

The invention claimed is:

1. A method of training a computer system for use in determining a transformation between coordinate frames of image data representing an imaged subject, the method comprising:
receiving first source image data representing a synthesized structure presented in a plurality of different views synthesized according to a model of the structure;
receiving second source image data representing a synthesized structure presented in a plurality of different views synthesized according to a model of the structure;
receiving third source image data representing a view of an imaged structure generated by an imaging apparatus in capturing an image of the subject;
in a first training operation, training a computer system, configured as a learning agent according to a machine learning algorithm, to determine a transformation between respective coordinate frames of at least two of said plurality of different views of the synthesized structure, using the labeled first source image data and labels associated therewith;
wherein the learning agent includes a domain classifier comprising a feature map generated by the learning agent during said first training operation and configured to generate therefrom a classification output indicating that received image data is one of said third source image data or said second source image data; and
in a second training operation, using the unlabeled second and third source image data without using labels associated therewith, training a computer system to determine a transformation between respective coordinate frames of said view of the imaged structure and said view of the synthesized structure, such that the ability of the domain classifier to discriminate between said synthesized structure and said imaged structure is reduced.

2. A method of training a computer system according to claim 1 in which the machine learning algorithm comprises one or more neural networks.

3. A method of training a computer system according to claim 1 in which the machine learning algorithm comprises:
a first neural network arranged for receiving input image data, and generating a first network output therefrom;
a second neural network arranged for receiving input image data, and generating a second network output therefrom;
a third neural network arranged for receiving as input both said first network output and said second network output, and generating a third network output therefrom for determining said transformation; and
a fourth neural network defining said domain classifier and arranged for receiving as input one or more of the activations generated by a layer of the first neural network and/or the second neural network.

4. A method of training a computer system according to claim 1 in which the first, second and fourth neural networks are each convolutional neutral networks (CNN), and the third neural network is a fully-connected neural network (FC).

5. A method of training a computer system according to claim 1 in which the machine learning algorithm comprises a domain-adversarial neural network.

6. A method of training a computer system according to claim 1 wherein said classification output is a probability estimate and said second training operation comprises training said computer system to determine said transformation such that said probability estimate approaches a value of 0.5 thereby reducing the ability of the domain classifier to discriminate between said synthesized structure and said imaged structure.

7. A method of training a computer system according to claim 6 in which said second training operation comprises training said computer system such that said probability estimate achieves a value of between 0.6 and 0.4.

8. A method of training a computer system according to claim 1 wherein said model is a three-dimensional model comprising three-dimensional image data, and the first target image data comprises two-dimensional image data representing a two-dimensional projection of the three-dimensional model to define a said view.

9. A method of training a computer system according to claim 1 , wherein said model is a three-dimensional model comprising three-dimensional image data, said first training operation comprising:
generating projection image data based on a two-dimensional projection of the model;
receiving, at the computer system, the projection image data as said first source image data;
determining, by the computer system, a reward for each of a plurality of actions applicable to the projection image data;
selecting an action based on the determined rewards; and
transforming the projection image data according to the selected action.

10. A method of training a computer system according to claim 9 , wherein said transforming the projection image data comprises:
applying the selected action to said model to generate a transformed model; and
generating further projection image data based on a two-dimensional projection of said transformed model.

11. A method of training a computer system according to claim 9 , wherein the reward for each of the plurality of actions is determined based on a spatial transformation of the projection image data.

12. A method of training a computer system according to claim 1 , wherein said model is a three-dimensional model comprising three-dimensional image data, said second training operation comprising:
generating projection image data based on a two-dimensional projection of the model;
receiving, at the computer system, the projection image data as said second source image data;
determining, by the computer system, a reward for each of a plurality of actions applicable to the projection image data;
selecting an action based on the determined rewards; and
transforming the projection image data according to the selected action.

13. A method of training a computer system according to claim 12 , wherein said transforming the projection image data comprises:
applying the selected action to said model to generate a transformed model; and
generating further projection image data based on a two-dimensional projection of said transformed model.

14. A method of training a computer system according to claim 12 , wherein the reward for each of the plurality of actions is determined based on a spatial transformation of the projection image data.

15. A method of training a computer system according claim 1 , wherein the third source image data comprises one of: magnetic resonance image data, computed tomography image data, and ultrasound image data, and X-ray image data.

16. A method of training a computer system according to claim 1 , determining a transformation between coordinate frames of image data representing an imaged subject, by:
receiving synthesized image data into the trained computer system, representing a view of a structure synthesized according to a model of the structure;
receiving captured image data into the trained computer system, representing an imaged structure generated by capturing an image of the subject; and
in the trained computer system, determining a transformation between the coordinate frames of said synthesized image data and said captured image data using said trained computer system.

17. A medical imaging device comprising:
a medical data acquisition scanner;
a processor that receives first source image data representing a synthesized structure presented in a plurality of different views synthesized according to a model of the structure;
said processor also receiving second source image data representing a synthesized struct

Assignees

• Siemens Healthcare Gmbh

Inventors

• Ceccaldi Pascal
• Kurzendorfer Tanja
• Mansi Tommaso
• Mountney Peter
• Piat Sebastien
• Toth Daniel

Classifications

• G06T12/00

  Tomographic reconstruction from projections · CPC title

• G06N3/048

  Activation functions · CPC title

• G06N3/006 Primary

  based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO] · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G06N3/047

  Probabilistic or stochastic networks · CPC title