Image segmentation and tracking based on statistical shape model

Patent metadata

Field	Value
Publication number	US-12327359-B2
Application number	US-202117538282-A
Country	US
Kind code	B2
Filing date	Nov 30, 2021
Priority date	Nov 30, 2021
Publication date	Jun 10, 2025
Grant date	Jun 10, 2025

Abstract

Described herein are systems, methods, and instrumentalities associated with segmenting and/or determining the shape of an anatomical structure. An artificial neural network (ANN) is used to perform these tasks based on a statistical shape model of the anatomical structure. The ANN is trained by evaluating and backpropagating multiple losses associated with shape estimation and segmentation mask generation. The model obtained using these techniques may be used for different clinical purposes including, for example, motion estimation and motion tracking.

First claim

What is claimed is:

1. An apparatus, comprising:
one or more processors configured to:
obtain an original three-dimensional (3D) representation of an anatomical structure, wherein the original 3D representation indicates a mean shape of the anatomical structure determined based on a statistical shape model of the anatomical structure;
obtain a two-dimensional (2D) medical scan image of the anatomical structure;
determine, based on the obtained 2D medical scan image and a pre-trained artificial neural network (ANN), a set of deformation parameters and a set of transformation parameters;
deform the original 3D representation of the anatomical structure using the set of deformation parameters to obtain a deformed 3D representation of the anatomical structure;
transform the deformed 3D representation of the anatomical structure using the set of transformation parameters to obtain a refined 3D representation of the anatomical structure; and
generate, using the pre-trained ANN, a segmentation of the anatomical structure based on the refined 3D representation of the anatomical structure, wherein the pre-trained ANN comprises one or more rendering layers configured to generate the segmentation in a differentiable manner, and wherein the pre-trained ANN further comprises a parameter prediction sub-network configured to predict the set of deformation parameters and the set of transformation parameters.

2. The apparatus of claim 1 , wherein the original 3D representation of the anatomical structure obtained by the one or more processors comprises a point cloud.

3. The apparatus of claim 1 , wherein the segmentation of the anatomical structure comprises a segmentation mask of the anatomical structure.

4. The apparatus of claim 1 , wherein the differentiable manner allows a gradient descent of a loss to be calculated and backpropagated during the training of the ANN.

5. The apparatus of claim 4 , wherein the loss is associated with the estimation of a segmentation mask during the training of the ANN.

6. The apparatus of claim 1 , wherein the one or more rendering layers of the pre-trained ANN being configured to generate the segmentation of the anatomical structure in the differentiable manner comprises the one or more rendering layers being configured to derive one or more samples of the segmentation via linear interpolation.

7. The apparatus of claim 1 , wherein the ANN is trained through a process that comprises:
obtaining a training image of the anatomical structure;
obtaining a 3D training representation of the anatomical structure that indicates the mean shape of the anatomical structure;
estimating values of the set of deformation parameters and the set of transformation parameters;
adjusting the 3D training representation of the anatomical structure using the estimated values of the set of deformation parameters and the set of transformation parameters;
predicting a segmentation of the anatomical structure based on the adjusted 3D training representation of the anatomical structure; and
adjusting parameters of the ANN based on a difference between the predicted segmentation of the anatomical structure and a ground truth segmentation of the anatomical structure.

8. The apparatus of claim 7 , wherein the parameters of the ANN are adjusted further based on a difference between the adjusted 3D training representation and a ground truth 3D representation of the anatomical structure.

9. The apparatus of claim 1 , wherein the one or more processors are further configured to track a motion of the anatomical structure using the set of deformation parameters and the set of transformation parameters.

10. A method for processing medical images, the method comprising:
obtaining an original three-dimensional (3D) representation of an anatomical structure, wherein the original 3D representation indicates a mean shape of the anatomical structure determined based on a statistical shape model of the anatomical structure;
obtaining a two-dimensional (2D) medical scan image of the anatomical structure;
determining, based on the obtained 2D medical scan image and a pre-trained artificial neural network (ANN), a set of deformation parameters and a set of transformation parameters;
deforming the original 3D representation of the anatomical structure using the set of deformation parameters to obtain a deformed 3D representation of the anatomical structure;
transforming the deformed 3D representation of the anatomical structure using the set of transformation parameters to obtain a refined 3D representation of the anatomical structure; and
generating, using the pre-trained ANN, a segmentation of the anatomical structure based on the refined 3D representation of the anatomical structure, wherein the pre-trained ANN comprises one or more rendering layers configured to generate the segmentation in a differentiable manner, and wherein the pre-trained ANN further comprises a parameter prediction sub-network configured to predict the set of deformation parameters and the set of transformation parameters.

11. The method of claim 10 , wherein the original 3D representation of the anatomical structure comprises a point cloud.

12. The method of claim 10 , wherein the segmentation of the anatomical structure comprises a segmentation mask of the anatomical structure.

13. The method of claim 10 , wherein the differentiable manner allows a gradient descent of a loss to be calculated and backpropagated during the training of the ANN.

14. The method of claim 13 , wherein the loss is associated with the estimation of a segmentation mask during the training of the ANN.

15. The method of claim 10 , wherein the one or more rendering layers of the pre-trained ANN being configured to generate the segmentation of the anatomical structure in the differentiable manner comprises the one or more rendering layers being configured to derive one or more samples of the segmentation via linear interpolation.

16. The method of claim 10 , wherein the ANN is trained through a process that comprises:
obtaining a training image of the anatomical structure;
obtaining a 3D training representation of the anatomical structure that indicates the mean shape of the anatomical structure;
estimating values of the set of deformation parameters and the set of transformation parameters;
adjusting the 3D training representation of the anatomical structure using the estimated values of the set of deformation parameters and the set of transformation parameters;
predicting a segmentation of the anatomical structure based on the adjusted 3D training representation of the anatomical structure; and
adjusting parameters of the ANN based on a difference between the predicted segmentation of the anatomical structure and a ground truth segmentation of the anatomical structure.

17. The method of claim 16 , wherein the parameters of the ANN are adjusted further based on a difference between the adjusted 3D training representation and a ground truth 3D representation of the anatomical structure.

18. The method of claim 10 , further comprising tracking a motion of the anatomical structure using the set of deformation parameters and the set of transformation parameters.

Assignees

• Shanghai United Imaging Intelligence Co Ltd

Inventors

• Chen Xiao
• Hu Xiaoling
• Chen Zhang
• Liu Yikang
• Chen Terrence
• Sun Shanhui

Classifications

• G06T3/18

  Image warping, e.g. rearranging pixels individually · CPC title

• G06T7/246

  using feature-based methods, e.g. the tracking of corners or segments · CPC title

• G06T2207/30048

  Heart; Cardiac · CPC title

• G06T2207/20124

  Active shape model [ASM] · CPC title

• G06T2207/20081

  Training; Learning · CPC title