System and method for one-shot anatomy localization with unsupervised vision transformers for three-dimensional (3d) medical images

Patent metadata

Field	Value
Publication number	US-2025104221-A1
Application number	US-202318491992-A
Country	US
Kind code	A1
Filing date	Oct 23, 2023
Priority date	Sep 27, 2023
Publication date	Mar 27, 2025
Grant date	—

Abstract

A method for performing one-shot anatomy localization includes obtaining a medical image of a subject. The method includes receiving a selection of both a template image and a region of interest within the template image, wherein the template image includes one or more anatomical landmarks assigned a respective anatomical label. The method includes inputting both the medical image and the template image into a trained vision transformer model. The method includes outputting from the trained vision transformer model both patch level features and image level features for both the medical image and the template image. The method still further includes interpolating pixel level features from the patch level features for both the medical image and the template image. The method includes utilizing the pixel level features within the region of interest of the template image to locate and label corresponding pixel level features in the medical image.

First claim

1 . A computer-implemented method for performing concurrent longitudinal lesion detection and analysis, comprising:
obtaining, at a processor, medical imaging data of a subject acquired at a later time point than reference medical imaging data acquired of the subject; automatically, via the processor, marking one or more non-lesion regions and one or more lesions in the medical imaging data based on one or more non-lesion regions and one or more lesions marked in the reference medical imaging data; inputting, via the processor, medical imaging data that has been marked into a trained vision transformer model; and outputting, via the processor, vision transformer features from the trained vision transformer model based on the marked one or more non-lesion regions and the one more lesions from the medical imaging data; identifying, via the processor, similar feature voxels in the medical imaging data based on the outputted vision transformer features to identify one or more lesions; and labeling, via the processor, the one or more lesions within the medical imaging data with segmentation masks.

2 . The computer-implemented method of claim 1 , further comprising:
obtaining, at the processor, the reference medical imaging data; receiving, at the processor, user inputs marking the one or more non-lesion regions and the one or more lesions of the reference medical imaging data; inputting, via the processor, the reference medical imaging data that has been marked into the trained vision transformer model; outputting, via the processor, vision transformer features from the trained vision transformer model based on the marked one or more non-lesions regions and the one or more lesions from the reference medical imaging data; identifying, via the processor, similar feature voxels in the reference medical imaging data based on the outputted vision transformer features based on the reference medical imaging data to identify the one or more lesions; and labeling, via the processor, the one or more lesions within the reference medical imaging data with the segmentation masks.

3 . The computer-implemented method of claim 2 , further comprising measuring, via the processor, one or more respective metrics of the one or more lesions in both the reference medical imaging data and the medical imaging data.

4 . The computer-implemented method of claim 3 , further comprising generating, via the processor, a report of changes in the respective metrics of the one or more lesions over time between the reference medical imaging data and the imaging data.

5 . The computer-implemented method of claim 2 , further comprising performing, via the processor, histogram matching between the reference imaging data and the medical imaging data prior to inputting the medical imaging data into the vision transformer model.

6 . The computer-implemented method of claim 1 , wherein identifying the similar voxel features comprises utilizing feature correlation or clustering or utilizing a compact machine learning model based on the user inputs.

7 . The computer-implemented method of claim 1 , further comprising automatically refining, via the processor, the segmentation masks.

8 . The computer-implemented method of claim 7 , wherein automatically refining the segmentation masks comprises utilizing a generalized segmentation refinement model that utilizes the identified similar voxel features.

9 . The computer-implemented method of claim 2 , further comprising receiving, at the processor, additional user inputs to update markings of the one or more non-lesion regions and the one or more lesions in the medical imaging data after the automatic marking of the one or more non-lesion regions and the one or more lesions in the medical imaging data.

10 . The computer-implemented method of claim 1 , wherein the trained vision transformer model was trained on a plurality of unlabeled medical images utilizing self-supervised learning.

11 . The computer-implemented method of claim 1 , wherein state information of the one or more lesions is always maintained in a database.

12 . The computer-implemented method of claim 1 , wherein the medical imaging data and the reference medical imaging data are not image registered with respect to each other.

13 . A system for performing concurrent longitudinal lesion detection and analysis, comprising:
a memory encoding processor-executable routines; and a processor configured to access the memory and to execute the processor-executable routines, wherein the processor-executable routines, when executed by the processor, cause the processor to:
obtain medical imaging data of a subject acquired at a later time point than reference medical imaging data acquired of the subject;
automatically mark one or more non-lesion regions and one or more lesions in the medical imaging data based on one or more non-lesion regions and one or more lesions marked in the reference medical imaging data;
input the medical imaging data that has been marked into a trained vision transformer model; and
output vision transformer features from the trained vision transformer model based on the marked one or more non-lesion regions and the one more lesions from the medical imaging data;
identify similar feature voxels in the medical imaging data based on the outputted vision transformer features to identify one or more lesions; and
label the one or more lesions within the medical imaging data with segmentation masks.

14 . The system of claim 13 , wherein the processor-executable routines, when executed by the processor further cause the processor to:
obtain the reference medical imaging data; receive user inputs marking the one or more non-lesion regions and the one or more lesions of the reference medical imaging data; input the reference medical imaging data that has been marked into the trained vision transformer model; output vision transformer features from the trained vision transformer model based on the marked one or more non-lesions regions and the one or more lesions from the reference medical imaging data; identify similar feature voxels in the reference medical imaging data based on the outputted vision transformer features based on the reference medical imaging data to identify the one or more lesions; and label the one or more lesions within the reference medical imaging data with the segmentation masks.

15 . The system of claim 14 , wherein the processor-executable routines, when executed by the processor further cause the processor to measure one or more respective metrics of the one or more lesions in both the reference medical imaging data and the medical imaging data.

16 . The system of claim 15 , wherein the processor-executable routines, when executed by the processor further cause the processor to generate a report of changes in the respective metrics of the one or more lesions over time between the reference medical imaging data and the imaging data.

17 . A non-transitory computer-readable medium, the computer-readable medium comprising processor-executable code that when executed by a processor, causes the processor to:
obtain medical imaging data of a subject acquired at a later time point than reference medical imaging data acquired of the subject; automatically mark one or more non-lesion regions and one or more lesions in the medical imaging data based on one or more non-lesion regions and one or more lesions marked in the reference medical imaging data; input the medical imaging data that has been marked into a trained vision transformer model; and

Assignees

• Ge Prec Healthcare Llc

Inventors

• Shanbhag Dattesh Dayanand
• Anand Deepa
• Mullick Rakesh
• Chatterjee Sudhanya
• Mongia Aanchal
• Patil Uday Damodar

Classifications

• G06T7/11

  Region-based segmentation · CPC title

• G06T7/0014

  using an image reference approach · CPC title

• G06V10/25 Primary

  Determination of region of interest [ROI] or a volume of interest [VOI] · CPC title

• G06V10/82

  using neural networks · CPC title

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title