Topogram prediction from surface data in medical imaging

Patent metadata

Field	Value
Publication number	US-11257259-B2
Application number	US-201816040998-A
Country	US
Kind code	B2
Filing date	Jul 20, 2018
Priority date	Aug 15, 2017
Publication date	Feb 22, 2022
Grant date	Feb 22, 2022

Abstract

For topogram predication from surface data, a sensor captures the outside surface of a patient. A generative adversarial network (GAN) generates the topogram representing an interior organ based on the outside surface of the patient. To further adapt to specific patients, internal landmarks are used in the topogram prediction. The topogram generated by one generator of the GAN may be altered based on landmarks generated by another generator.

First claim

What is claimed is:

1. A method for obtaining a predicted synthetic topogram from surface data in a medical imaging system, the method comprising:
capturing, with a sensor, the surface data, wherein the surface data represents an outer surface of a patient;
generating, by an image processor, the predicted synthetic topogram by a machine-learned generative adversarial network in response to input of the surface data captured by the sensor to the machine-learned generative adversarial network, wherein the predicted synthetic topogram includes an internal organ of the patient, and wherein the predicted synthetic topogram represents a two-dimensional projection of the internal organ of the patient; and
displaying, by a display device, the predicted synthetic topogram.

2. The method of claim 1 wherein capturing comprises capturing with the sensor being a depth sensor.

3. The method of claim 1 wherein capturing comprises capturing with the sensor being a camera where the surface data based on optical measurements.

4. The method of claim 1 wherein generating further comprises fitting a statistical shape model to the output of the sensor for the outer surface, the surface data comprising (a) depths from the fit statistical shape model and (b) a projection of the outer surface or a thickness.

5. The method of claim 1 wherein generating comprises generating with the machine-learned generative adversarial network comprising a fully convolutional network with an encoder and a decoder.

6. The method of claim 5 wherein the encoder and the decoder comprise a U-Net with skip connections between the encoder and the decoder.

7. The method of claim 1 wherein generating comprises generating with the machine-learned generative adversarial network being a stacked U-Net.

8. The method of claim 1 wherein generating comprises generating with the machine-learned generative adversarial network having been trained with an L1 loss.

9. The method of claim 1 wherein generating comprises generating with the machine-learned generative adversarial network including a first generator for generating a first X-ray image from the surface data, a second generator for generating a landmark image from the first X-ray image or a second X-ray image and the surface data, and a third generator for generating the second X-ray image from the surface data and the landmark image, the second X-ray image comprising the predicted synthetic topogram.

10. The method of claim 9 wherein generating comprises iteratively using the second and third generators.

11. The method of claim 9 further comprising:
receiving user input of a landmark location of a first landmark;
wherein generating comprises generating the predicted synthetic topogram with the first landmark constrained to the landmark location.

12. The method of claim 1 wherein generating comprises generating with the generative adversarial network having been trained with a gradient penalty.

13. The method of claim 1 further comprising:
configuring a medical scanner based on the predicted synthetic topogram; and
imaging, by the medical scanner, the patient as configured based on the predicted synthetic topogram.

14. A method for estimated synthetic topogram prediction from surface data in a medical imaging system, the method comprising:
capturing, with a sensor, an outer surface of a patient;
generating, by an image processor, the estimated synthetic topogram by first and second machine-learned networks in response to input of the surface data captured by the sensor to the first and second machine-learned networks, the surface data being from an output of the sensor for the outer surface, the estimated synthetic topogram representing an internal organ of the patient as a projection through the patient, the first machine-learned network outputting a spatial marker map, and the second machine-learned network outputting the estimated synthetic topogram based on the surface data and the spatial marker map, wherein the estimated synthetic topogram includes an internal organ of the patient, and wherein the estimated synthetic topogram represents a two-dimensional projection of the internal organ of the patient; and
displaying, by a display device, the estimated synthetic topogram.

15. The method of claim 14 wherein generating further comprises iteratively using the first and second machine-learned networks, the first machine-learned network outputting the spatial marker map in response to input of the estimated synthetic topogram and the surface data, and wherein displaying comprises displaying a final one of the estimated synthetic topograms from the iterations.

16. The method of claim 15 wherein generating comprises generating with a third machine-learned network, the third machine-learned network outputting an initial estimated synthetic topogram for the iterating to the first machine-learned for an initial iteration.

17. The method of claim 14 wherein generating comprises generating with the first and second machine-learned networks comprising Wasserstein generative adversarial networks having been trained with gradient penalties.

18. A medical imaging system for estimated synthetic topogram prediction, the medical imaging system comprising:
a depth sensor configured to measure depths from the sensor to a patient;
an image processor configured to apply a machine-learned generator of a trained generative adversarial network to depth information from the depths, the machine-learned generator having been trained to generate the estimated synthetic topogram from the depth information, wherein the estimated synthetic topogram includes an internal organ of the patient, and wherein the estimated synthetic topogram represents a two-dimensional projection of the internal organ of the patient; and
a display configured to display the estimated synthetic topogram.

19. The medical imaging system of claim 18 wherein the generative adversarial network comprises a Wasserstein generative adversarial network having been trained using a gradient penalty.

20. The medical imaging system of claim 18 wherein the machine-learned generator comprises first and second generators, the first generator configured to output a landmark probability map and the second generator configured to output the estimated synthetic topogram based on input of the depth information and the landmark probability map.

Assignees

• Siemens Healthcare Gmbh

Inventors

• Teixeira Brian
• Singh Vivek Kumar
• Tamersoy Birgi
• Chen Terrence
• Ma Kai
• Krauss Andreas

Classifications

• G06V10/82

  using neural networks · CPC title

• G06V10/776

  Validation; Performance evaluation · CPC title

• G06V10/764

  using classification, e.g. of video objects · CPC title

• G06T12/00

  Tomographic reconstruction from projections · CPC title

• G06N3/047

  Probabilistic or stochastic networks · CPC title