Determining malignancy of pulmonary nodules using deep learning

Patent metadata

Field	Value
Publication number	US-11244453-B2
Application number	US-201816179154-A
Country	US
Kind code	B2
Filing date	Nov 2, 2018
Priority date	Nov 2, 2018
Publication date	Feb 8, 2022
Grant date	Feb 8, 2022

Abstract

Systems and method are described for determining a malignancy of a nodule. A medical image of a nodule of a patient is received. A patch surrounding the nodule is identified in the medical image. A malignancy of the nodule in the patch is predicted using a trained deep image-to-image network.

First claim

The invention claimed is:

1. A method for determining a malignancy of a nodule, comprising:
receiving a medical image of a nodule of a patient;
defining a patch surrounding the nodule in the medical image;
predicting a malignancy of the nodule in the patch using a trained deep image-to-image network comprising an encoder and a decoder;
comparing the predicted malignancy of the nodule in the patch with one or more thresholds;
in response to the comparing of the predicted malignancy of the nodule in the patch with the one or more thresholds:
receiving a second medical image of the nodule, the second medical image being of a different modality than the medical image and more detailed than the medical image;
defining a second patch surrounding the nodule in the second medical image; and
predicting a malignancy of the nodule in the second patch using the trained deep image-to-image network,

wherein the trained deep image-to-image network is trained to, based on low level representations of training patches determined by the encoder, 1) reconstruct the training patches by a first branch of layers of the decoder and 2) predict a malignancy of nodules in the training patches by a second branch of layers of the decoder, and
wherein the second branch of layers of the decoder is used to perform the predicting of the malignancy of the nodule in the patch without using the first branch of layers of the decoder.

2. The method of claim 1 , further comprising:
training the trained deep image-to-image network using the training images and results of a histopathological examination of the nodules in the training images.

3. The method of claim 2 , wherein training the trained deep image-to-image network using the training images and results of a histopathological examination of the nodules in the training images further comprises:
training the trained deep image-to-image network using additional training images and results of a radiologist examination of nodules in the additional training images.

4. The method of claim 1 , wherein predicting a malignancy of the nodule in the patch using a trained deep image-to-image network comprises:
determining a score indicating the malignancy of the nodule.

5. The method of claim 1 , wherein predicting a malignancy of the nodule in the patch using a trained deep image-to-image network comprises:
classifying the nodule as malignant or not malignant.

6. The method of claim 1 , wherein predicting a malignancy of the nodule in the patch using a trained deep image-to-image network comprises:
inputting the patch into the encoder for converting the patch to a low level representation; and
predicting the malignancy of the nodule in the patch from the low level representation by the decoder.

7. The method of claim 1 , wherein the trained deep image-to-image network comprises a trained deep reasoner network.

8. An apparatus for determining a malignancy of a nodule, comprising:
means for receiving a medical image of a nodule of a patient;
means for defining a patch surrounding the nodule in the medical image; and
means for predicting a malignancy of the nodule in the patch using a trained deep image-to-image network comprising an encoder and a decoder;
means for comparing the predicted malignancy of the nodule in the patch with one or more thresholds;
in response to the comparing of the predicted malignancy of the nodule in the patch with the one or more thresholds:
means for receiving a second medical image of the nodule, the second medical image being of a different modality than the medical image and more detailed than the medical image;
means for defining a second patch surrounding the nodule in the second medical image; and
means for predicting a malignancy of the nodule in the second patch using the trained deep image-to-image network,

wherein the trained deep image-to-image network is trained to, based on low level representations of training patches determined by the encoder, 1) reconstruct the training patches by a first branch of layers of the decoder and 2) predict a malignancy of nodules in the training patches by a second branch of layers of the decoder, and
wherein the second branch of layers of the decoder is used to perform the predicting of the malignancy of the nodule in the patch without using the first branch of layers of the decoder.

9. The apparatus of claim 8 , further comprising:
means for training the trained deep image-to-image network using the training images and results of a histopathological examination of the nodules in the training images.

10. The apparatus of claim 9 , wherein the means for training the trained deep image-to-image network using the training images and results of a histopathological examination of the nodules in the training images further comprises:
means for training the trained deep image-to-image network using additional training images and results of a radiologist examination of nodules in the additional training images.

11. The apparatus of claim 8 , wherein the means for predicting a malignancy of the nodule in the patch using a trained deep image-to-image network comprises:
means for determining a score indicating the malignancy of the nodule.

12. The apparatus of claim 8 , wherein the means for predicting a malignancy of the nodule in the patch using a trained deep image-to-image network comprises:
means for classifying the nodule as malignant or not malignant.

13. A non-transitory computer readable medium storing computer program instructions for determining a malignancy of a nodule, the computer program instructions when executed by a processor cause the processor to perform operations comprising:
receiving a medical image of a nodule of a patient;
defining a patch surrounding the nodule in the medical image;
predicting a malignancy of the nodule in the patch using a trained deep image-to-image network comprising an encoder and a decoder;
comparing the predicted malignancy of the nodule in the patch with one or more thresholds;
in response to the comparing of the predicted malignancy of the nodule in the patch with the one or more thresholds:
receiving a second medical image of the nodule, the second medical image being of a different modality than the medical image and more detailed than the medical image;
defining a second patch surrounding the nodule in the second medical image; and
predicting a malignancy of the nodule in the second patch using the trained deep image-to-image network,

wherein the trained deep image-to-image network is trained to, based on low level representations of training patches determined by the encoder, 1) reconstruct the training patches by a first branch of layers of the decoder and 2) predict a malignancy of nodules in the training patches by a second branch of layers of the decoder, and
wherein the second branch of layers of the decoder is used to perform the predicting of the malignancy of the nodule in the patch without using the first branch of layers of the decoder.

14. The non-transitory computer readable medium of claim 13 , the operations further comprising:
training the trained deep image-to-image network using the training images and results of a histopathological examination of the nodules in the training images.

15. The non-transitory computer readable medium of claim 13 , wherein predicting a malignancy of the nodule in the patch using a trained deep image-to-image network comprises:
inputting the patch into the encoder for converting the patch to a low level representation; and
predicting the malignancy of the nodule in the patch from the

Assignees

• Siemens Healthcare Gmbh

Inventors

• Grbic Sasa
• Comaniciu Dorin
• Georgescu Bogdan
• Liu Siqi
• Ionasec Razvan

Classifications

• G06T7/0012 Primary

  Biomedical image inspection · CPC title

• G06T7/0014 Primary

  using an image reference approach · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G06F18/211

  Selection of the most significant subset of features · CPC title

• G06F18/24

  Classification techniques · CPC title