Systems and methods for tumor characterization

Patent metadata

Field	Value
Publication number	US-11282193-B2
Application number	US-202016836855-A
Country	US
Kind code	B2
Filing date	Mar 31, 2020
Priority date	Mar 31, 2020
Publication date	Mar 22, 2022
Grant date	Mar 22, 2022

Abstract

Systems and methods for characterizing a region of interest (ROI) in a medical image are provided. An exemplary system may include a memory storing instructions and at least one processor communicatively coupled to the memory to execute the instructions which, when executed by the processor, may cause the processor to perform operations. The operations may include detecting one or more candidate ROIs from the medical image using a three-dimensional (3D) machine learning network. The operations may also include determining a key slice for each candidate ROI. The operations may further include selecting a primary ROI from the one or more candidate ROIs based on the respective key slices. In addition, the operations may include classifying the primary ROI into one of a plurality of categories using a texture-based classifier based on the key slice corresponding to the primary ROI.

First claim

The invention claimed is:

1. A system for characterizing a region of interest (ROI) in a medical image, comprising:
a memory storing computer-readable instructions; and
at least one processor communicatively coupled to the memory to execute the computer-readable instructions, wherein the computer-readable instructions, when executed by the at least one processor, cause the at least one processor to perform operations comprising:
detecting one or more candidate ROIs from the medical image using a three-dimensional (3D) machine learning network;
determining a key slice for each candidate ROI;
selecting a primary ROI from the one or more candidate ROIs based on the respective key slices; and
classifying the primary ROI into one of a plurality of categories using a texture-based classifier based on the key slice corresponding to the primary ROI, wherein the texture-based classifier is spatially adaptive with a set of spatially adaptive factors generated based on a tumor mask.

2. The system of claim 1 , wherein the primary ROI comprises an image of a liver tumor.

3. The system of claim 1 , wherein the plurality of categories comprise at least two from a group consisting of an intrahepatic cholangiocarcinoma (ICC), a hepatocellular carcinoma (HCC), a metastasis, and a benign tumor.

4. The system of claim 1 , wherein the medical image comprises at least one image from a set of multi-phase Computed Tomography (CT) images.

5. The system of claim 1 , wherein detecting the one or more candidate ROIs comprises:
determining a 3D bounding box for each of the one or more candidate ROIs, wherein the 3D bounding box encloses the corresponding candidate ROI.

6. The system of claim 5 , wherein determining the key slice comprises:
segmenting each of the one or more candidate ROIs in the corresponding 3D bounding box to separate, in a plurality of two-dimensional (2D) slices of the medical image, a cross-section of that candidate ROI from its surroundings; and
selecting, for each of the one or more candidate ROIs, a 2D slice of the medical image in which the corresponding cross-section has a largest area as the key slice for that candidate ROI.

7. The system of claim 1 , wherein selecting the primary ROI comprises:
classifying, by an image-based classifier, the candidate ROIs into a first group comprising the primary ROI and a second group comprising one or more non-primary ROIs based on the respective key slices of the candidate ROIs.

8. The system of claim 1 , wherein classifying the primary ROI into one of the plurality of categories comprises:
receiving a set of visual descriptors indicating features in the key slice corresponding to the primary ROI;
determining a set of residuals by comparing the set of visual descriptors with a set of codewords;
assigning a corresponding weight to each residual so that a set of weighted residuals is generated, wherein the corresponding weight is determined by a softmax function using learnable smoothing factors; and
aggregating the set of weighted residuals to generate a global feature indicating an overall texture based on the set of spatially adaptive factors.

9. The system of claim 8 , wherein aggregating the set of weighted residuals to generate the global feature indicating the overall texture based on the set of spatially adaptive factors comprises:
applying a corresponding spatially adaptive factor from the set of spatially adaptive factors to each weighted residual to generate one or more spatially adaptive weighted residuals; and
aggregating the one or more spatially adaptive weighted residuals to generate the global feature indicating the overall texture.

10. The system of claim 1 , wherein the 3D machine learning network is trained by a training data set comprising annotations to pathologically confirmed tumors, wherein at least part of the training data set is harvested from unlabeled medical image data.

11. The system of claim 10 , wherein:
the unlabeled medical image data comprises unlabeled multi-phase Computed Tomography (CT) image data; and
at least part of the training data set is harvested from the unlabeled multi-phase CT image data by a data curation process comprising:
preliminarily training the 3D machine learning network using both multi-phase CT image data and single-phase CT image data, wherein the multi-phase CT image data are is input as multiple single-phase CT image data;
applying the preliminarily trained 3D machine learning network to the unlabeled multi-phase CT image data to obtain 3D bounding boxes of predicted ROIs in each individual phase;
merging corresponding 3D bounding boxes obtained in individual phases into a combined 3D bounding box for each predicted ROI; and
verifying the combined 3D bounding box for the corresponding predicted ROI through a quality assurance process.

12. The system of claim 1 , wherein an output of the 3D machine learning network comprises a 3D heatmap comprising a first value at tumor centers and a second value at other locations.

13. A method for characterizing a region of interest (ROI) in a medical image, comprising:
detecting, by a processor, one or more candidate ROIs from the medical image using a three-dimensional (3D) machine learning network;
determining, by the processor, a key slice for each candidate ROI;
selecting, by the processor, a primary ROI from the one or more candidate ROIs based on the respective key slices; and
classifying, by the processor, the primary ROI into one of a plurality of categories using a texture-based classifier based on the key slice corresponding to the primary ROI, wherein the texture-based classifier is spatially adaptive with a set of spatially adaptive factors generated based on a tumor mask.

14. The method of claim 13 , wherein detecting the one or more candidate ROIs comprises:
determining a 3D bounding box for each of the one or more candidate ROIs, wherein the 3D bounding box encloses the corresponding candidate ROI.

15. The method of claim 14 , wherein determining the key slice comprises:
segmenting each of the one or more candidate ROIs in the corresponding 3D bounding box to separate, in a plurality of two-dimensional (2D) slices of the medical image, a cross-section of that candidate ROI from its surroundings; and
selecting, for each of the one or more candidate ROIs, a 2D slice of the medical image in which the corresponding cross-section has a largest area as the key slice for that candidate ROI.

16. The method of claim 13 , wherein selecting the primary ROI comprises:
classifying, by an image-based classifier, the candidate ROIs into a first group comprising the primary ROI and a second group comprising one or more non-primary ROIs based on the respective key slices of the candidate ROIs.

17. The method of claim 13 , wherein classifying the primary ROI into one of the plurality of categories comprises:
receiving a set of visual descriptors indicating features in the key slice corresponding to the primary ROI;
determining a set of residuals by comparing the set of visual descriptors with a set of codewords;
assigning corresponding a weight to each residual so that a set of weighted residuals is generated, wherein the corresponding weight is determined by a softmax function using learnable smoothing factors; and
aggregating the set of weighted residuals to generate a global feature indicating an overall texture based on the set of spatially adaptive factors.

18. The method of claim 17 , wherein aggregating the set of weighted residuals to generate the global feature indi

Assignees

• Ping An Tech Shenzhen Co Ltd

Inventors

• Harrison Adam P
• Huo Yuankai
• Cai Jinzheng
• Raju Ashwin
• Yan Ke
• Lu Le

Classifications

• G06V20/647

  by matching two-dimensional images to three-dimensional objects · CPC title

• G06V10/764

  using classification, e.g. of video objects · CPC title

• G06F18/2431

  Multiple classes · CPC title

• G06F18/24

  Classification techniques · CPC title

• G06T2207/30096

  Tumor; Lesion · CPC title