Trackerless 2D ultrasound frame to 3D image volume registration

Patent metadata

Field	Value
Publication number	US-12367597-B2
Application number	US-202117793215-A
Country	US
Kind code	B2
Filing date	Jan 15, 2021
Priority date	Jan 15, 2020
Publication date	Jul 22, 2025
Grant date	Jul 22, 2025

Abstract

One embodiment provides an apparatus for registering a two dimensional (2D) ultrasound (US) frame and a three dimensional (3D) magnetic resonance (MR) volume. The apparatus includes a first deep neural network (DNN) and an image fusion management circuitry. The first DNN is configured to determine a 2D US pose vector based, at least in part, on 2D US frame data. The image fusion management circuitry is configured to register the 2D US frame data and a 3D MR volume data. The registering is based, at least in part, on the 2D US pose vector.

First claim

What is claimed is:

1. An apparatus for registering a two dimensional (2D) ultrasound (US) frame and a three dimensional (3D) magnetic resonance (MR) volume, the apparatus comprising:
a first deep neural network (DNN) configured to determine a 2D US pose vector based, at least in part, on 2D US frame data and 3D MR volume data, the pose vector being configured to provide relative position and orientation of the 2D US frame with respect to the MR volume; and
an image fusion management circuitry configured to register the 2D US frame data and the 3D MR volume data, the registering based, at least in part, on the 2D US pose vector.

2. The apparatus of claim 1 , further comprising a second DNN configured to determine a 3D US pose vector based, at least in part, on the 2D US frame data;
a 3D US volume reconstruction circuitry configured to reconstruct a 3D US volume based, at least in part, on the 3D US pose vector; and
a third DNN configured to determine a transformation relating the 3D MR volume and the 3D US volume,
wherein the registering of the 2D US frame data and the 3D MR volume data is based, at least in part, on the transformation.

3. The apparatus of claim 2 , wherein the first DNN and the second and third DNNs are trained alternatingly.

4. The apparatus of claim 2 , wherein the third DNN comprises a plurality of DNN stages configured for coarse-to-fine multi-stage registration.

5. The apparatus of claim 2 , wherein the third DNN is configured as a generative adversarial network (GAN).

6. The apparatus of claim 1 , wherein the first DNN is initially trained using population data and is subsequently trained using patient data.

7. A method for registering a two dimensional (2D) ultrasound (US) frame and a three dimensional (3D) magnetic resonance (MR) volume, the method comprising:
determining, by a first deep neural network (DNN), a 2D US pose vector based, at least in part, on 2D US frame data and 3D MR volume data, the pose vector providing relative position and orientation of the 2D US frame with respect to the MR volume; and
registering, by an image fusion management circuitry, the 2D US frame data and the 3D MR volume data, the registering based, at least in part, on the 2D US pose vector.

8. The method of claim 7 , further comprising determining, by a second DNN, a 3D US pose vector based, at least in part, on the 2D US frame data;
reconstructing, by a 3D US volume reconstruction circuitry, a 3D US volume based, at least in part, on the 3D US pose vector; and
determining, by a third DNN, a transformation relating the 3D MR volume and the 3D US volume,
wherein the registering of the 2D US frame data and the 3D MR volume data is based, at least in part on the transformation.

9. The method of claim 8 , wherein the first DNN and the second and third DNNs are trained alternatingly.

10. The method of claim 8 , wherein the third DNN comprises a plurality of DNN stages configured for coarse-to-fine multi-stage registration.

11. The method of claim 8 , wherein the third DNN is configured as a generative adversarial network (GAN).

12. The method of claim 7 , wherein the first DNN is initially trained using population data.

13. The method of claim 12 , wherein the first DNN is trained using patient data.

14. A system for registering a two dimensional (2D) ultrasound (US) frame and a three dimensional (3D) magnetic resonance (MR) volume, the system comprising:
a processor;
a memory;
input/output circuitry;
a first deep neural network (DNN) configured to determine a 2D US pose vector based, at least in part, on 2D US frame data and 3D MR volume data, the pose vector being configured to provide relative position and orientation of the 2D US frame with respect to the MR volume; and
an image fusion management circuitry configured to register the 2D US frame data and the 3D MR volume data, the registering based, at least in part, on the 2D US pose vector.

15. The system of claim 14 , further comprising a second DNN configured to determine a 3D US pose vector based, at least in part, on the 2D US frame data;
a 3D US volume reconstruction circuitry configured to reconstruct a 3D US volume based, at least in part, on the 3D US pose vector; and
a third DNN configured to determine a transformation relating the 3D MR volume and the 3D US volume,
wherein the registering of the 2D US frame data and the 3D MR volume data is based, at least in part on the transformation.

16. The system of claim 15 , wherein the first DNN and the second and third DNNs are trained alternatingly.

17. The system of claim 15 , wherein the third DNN comprises a plurality of DNN stages configured for coarse-to-fine multi-stage registration.

18. The system of claim 15 , wherein the third DNN is configured as a generative adversarial network (GAN).

19. The system of claim 14 , wherein the first DNN is initially trained using population data and is subsequently trained using patient data.

20. A computer readable storage device having stored thereon instructions that when executed by one or more processors result in the following operations comprising:
the method according to any one of claims 7 through 13 .

Assignees

• Rensselaer Polytech Inst

Inventors

• Yan Pingkun

Classifications

• G06T2207/30081

  Prostate · CPC title

• G06T2207/20221

  Image fusion; Image merging · CPC title

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title

• G06T2207/20081

  Training; Learning · CPC title

• G06T2207/20016

  Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform · CPC title