Method, device, and computer program product for deep lesion tracker for monitoring lesions in four-dimensional longitudinal imaging

Patent metadata

Field	Value
Publication number	US-11410309-B2
Application number	US-202117213804-A
Country	US
Kind code	B2
Filing date	Mar 26, 2021
Priority date	Dec 3, 2020
Publication date	Aug 9, 2022
Grant date	Aug 9, 2022

Abstract

The present disclosure provides a computer-implemented method, a device, and a computer program product for deep lesion tracker. The method includes inputting a search image into a first three-dimensional DenseFPN (feature pyramid network) of an image encoder and inputting a template image into a second three-dimensional DenseFPN of the image encoder to extract image features; encoding anatomy signals of the search image and the template image as Gaussian heatmaps, and inputting the Gaussian heatmap of the template image into a first anatomy signal encoders (ASE) and inputting the Gaussian heatmap of the search image into a second ASE to extract anatomy features; inputting the image features and the anatomy features into a fast cross-correlation layer to generate correspondence maps, and computing a probability map according to the correspondence maps; and performing supervised learning or self-supervised learning to predict a lesion center in the search image.

First claim

What is claimed is:

1. A deep lesion tracker method for medical images, the method comprising:
providing an image pair including a search image and a template image;
inputting the search image into a first three-dimensional DenseFPN (feature pyramid network) of an image encoder, and inputting the template image into a second three-dimensional DenseFPN of the image encoder to extract image features of the search image and the template image in three different scales, wherein the first and second three-dimensional DenseFPNs are configured with shared weights;
encoding anatomy signals of the search image and the template image as Gaussian heatmaps centered at lesion locations, inputting the Gaussian heatmap of the template image into a first anatomy signal encoders (ASE) and inputting the Gaussian heatmap of the search image into a second ASE to extract anatomy features of the search image and the template image in three different scales, wherein the first and the second ASEs are configured with shared weights;
inputting the image features and the anatomy features into a fast cross-correlation layer to generate correspondence maps, and computing a probability map according to the correspondence maps; and
performing supervised learning or self-supervised learning to predict a lesion center in the search image.

2. The method according to claim 1 , wherein encoding the anatomy signals as the Gaussian heatmaps includes:
for the template image, using a location and a size of a template lesion to compute the anatomy signals of the template image; and
for the search image, using an affine-projected location and an affine-projected size of the template lesion to compute the anatomy signals of the search image.

3. The method according to claim 2 , wherein inputting the image features and the anatomy features into the fast cross-correlation layer to generate the correspondence maps includes:
fusing the image features of the template image and the anatomy features of the template image.

4. The method according to claim 3 , wherein after fusing the image features of the template image and the anatomy features of the template image, the method further includes:
defining a cropping function to extract a template kernel K and another template kernel K g , wherein:
a size of the template kernel K g is greater than a size of the template kernel K; and the template kernel K g is decomposed into kernels K g,x , K g,y , and K g,z , along axial, coronal, and sagittal directions, respectively.

5. The method according to claim 4 , wherein a correspondence map is computed by:

M
=

(

K
*
S

)

+

(

∑

i
∈
x

,
y
,
z

⁢

K

g
,
i

*
S

)

wherein + denotes element-wise sum, * denotes multiplication, S=ψ(I s )⊙ϕ(G s ), I s , I s is the search image, G s is an anatomy signal map of the search image, ⊙ denotes element-wise multiplication, ψ and ϕ denote network encoders that generate image features and anatomy features, respectively.

6. The method according to claim 5 , wherein after computing the correspondence map, the method further includes:
determining the lesion center in the search image according to the probability map computed based on the correspondence maps.

7. A deep lesion tracker device for medical images comprising:
a memory, containing a computer program stored thereon; and
a processor, coupled with the memory and configured, when the computer program being executed, to perform a method including:
providing an image pair including a search image and a template image;
inputting the search image into a first three-dimensional DenseFPN (feature pyramid network) of an image encoder and inputting the template image into a second three-dimensional DenseFPN of the image encoder to extract image features of the search image and the template image in three different scales, wherein the first and second three-dimensional DenseFPNs are configured with shared weights;
encoding anatomy signals of the search image and the template image as Gaussian heatmaps centered at lesion locations, and inputting the Gaussian heatmap of the template image into a first anatomy signal encoders (ASE) and inputting the Gaussian heatmap of the search image into a second ASE to extract anatomy features of the search image and the template image in three different scales, wherein the first and the second ASEs are configured with shared weights;
inputting the image features and the anatomy features into a fast cross-correlation layer to generate correspondence maps, and computing a probability map according to the correspondence maps; and
performing supervised learning or self-supervised learning to predict a lesion center in the search image.

8. The device according to claim 7 , wherein encoding the anatomy signals as the Gaussian heatmaps includes:
for the template image, using a location and a size of a template lesion to compute the anatomy signals of the template image; and
for the search image, using an affine-projected location and an affine-projected size of the template lesion to compute the anatomy signals of the search image.

9. The device according to claim 8 , wherein inputting the image features and the anatomy features into the fast cross-correlation layer to generate the correspondence maps includes:
fusing the image features of the template image and the anatomy features of the template image.

10. The device according to claim 9 , wherein after fusing the image features of the template image and the anatomy features of the template image, the method further includes:
defining a cropping function to extract a template kernel K and another template kernel K g , wherein:
a size of the template kernel K g is greater than a size of the template kernel K; and the template kernel K g is decomposed into kernels K g,x , K g,y and K g,z , along axial, coronal, and sagittal directions, respectively.

11. The device according to claim 10 , wherein a correspondence map is computed by:

M
=

(

K
*
S

)

+

(

∑

i
∈
x

,

Assignees

• Ping An Tech Shenzhen Co Ltd

Inventors

• Cai Jinzheng
• Tang Youbao
• Yan Ke
• Harrison Adam P
• Lu Le

Classifications

• G06N3/045

  Combinations of networks · CPC title

• G16H30/40

  for processing medical images, e.g. editing · CPC title

• G06N3/088

  Non-supervised learning, e.g. competitive learning · CPC title

• G16H50/20

  for computer-aided diagnosis, e.g. based on medical expert systems · CPC title

• G06N3/09

  Supervised learning · CPC title