Content-based medical image rendering based on machine learning

Patent metadata

Field	Value
Publication number	US-10339695-B2
Application number	US-201715643973-A
Country	US
Kind code	B2
Filing date	Jul 7, 2017
Priority date	Mar 10, 2016
Publication date	Jul 2, 2019
Grant date	Jul 2, 2019

Abstract

An artificial intelligence agent is machine trained and used to provide physically-based rendering settings. By using deep learning and/or other machine training, settings of multiple rendering parameters may be provided for consistent imaging even in physically-based rendering.

First claim

We claim:

1. A method for content-based rendering based on machine learning in a rendering system, the method comprising:
loading, from memory, a medical dataset representing a three-dimensional region of a patient;
applying, by a machine, the medical dataset to a machine-learnt model, the machine-learned model trained with deep learning to extract features from the medical dataset and trained to output values for two or more volume rendering parameters that correspond to input of the medical dataset, the two or more volume rendering parameters being settings of a volume renderer, the settings used by the volume renderer to control rendering, from three dimensions to two-dimensions, an image of the three-dimensional region of the patient;
rendering, by the volume renderer, the image of the three-dimensional region of the patient from the medical dataset using the output values resulting from the applying as the settings to control the rendering from the medical dataset, the rendering of the medical dataset of the three-dimensional region being to the image in the two-dimensions; and
transmitting the image.

2. The method of claim 1 further comprising loading patient information other than the medical dataset representing the three-dimensional region of the patient.

3. The method of claim 1 wherein applying comprises applying to output the values for the two or more volume rendering parameters as all of controls for data consistency handling, lighting design, material propriety, and internal renderer property based on applying user input of viewing design and the medical dataset.

4. The method of claim 1 wherein applying comprises applying with the machine-learnt model trained to output the values resulting in the image corresponding to a standard image despite differences in the medical dataset.

5. The method of claim 1 wherein applying comprises applying with the machine-learnt model comprising a regression, classification, or reinforcement learnt model.

6. The method of claim 1 wherein applying comprises applying with the deep learning as a deep neural network.

7. The method of claim 1 wherein applying comprises applying with the machine-learnt model as a deep reinforcement learnt model.

8. The method of claim 1 wherein rendering comprises rendering with unbiased path tracing.

9. The method of claim 1 wherein transmitting comprises transmitting as part of a diagnostic report, as an initial image of an interactive viewing, or as an overlay in augmented reality.

10. The method of claim 1 further comprising:
measuring ambient light with a light sensor;
wherein applying comprises applying the ambient light and the medical dataset, the machine-learnt model trained to output the values based in part on the ambient light.

11. A method for machine training for content-based rendering in a machine training system, the method comprising:
inputting first volume data of a volume of a patient, a first image of the volume, and first values of rendering parameters to training of an artificial intelligence, rendering parameters being settings to control rendering from the volume to a two-dimensional image;
machine training, with a machine, the artificial intelligence to output second values of the rendering parameters for second volume data where the second values control the rendering from the volume to provide a second rendered image of the second volume data modeled on the first image; and
storing the trained artificial intelligence.

12. The method of claim 11 wherein inputting comprises inputting the patient non-image information to the training.

13. The method of claim 11 wherein training comprises training the artificial intelligence to output the second values as two or more of data consistency, transfer function, lighting, and viewing parameters.

14. The method of claim 11 wherein training comprises training the artificial intelligence to output the second values based on user selected viewing camera parameters, the rendering parameters for which second values are to be output being other than the viewing camera parameters.

15. The method of claim 11 wherein training so the second rendered image is modeled after the first rendered image comprises training with a metric of similarity between the first rendered image and the second rendered image.

16. The method of claim 11 wherein the first image comprises a reference photograph or video of the patient.

17. The method of claim 11 wherein inputting comprises perturbing the rendering parameters, creating a collection of sets of the rendering parameters, and wherein training comprises training based on selection of a sub-set of the sets.

18. The method of claim 17 wherein selection of the sub-set comprises selection by a user based on visual examination of images rendered using the sets.

19. The method of claim 11 wherein machine training comprises deep learning with regression, classification, or reinforcement learning.

20. The method of claim 19 wherein machine training comprises deep reinforcement learning with a similarity of the second rendered image to the first image as a reinforcement.

21. The method of claim 20 wherein deep reinforcement learning comprises selecting with a probability distribution of different similarities including the similarity.

22. A system for content-based rendering based on machine learning, the system comprising:
a medical scanner configured to scan a patient;
a machine configured to output settings for rendering parameters by application of data from the scan to a machine-learnt model, the rendering parameters being controls for performing rendering from a volume to a two-dimensional image, and the settings learned to provide a first image from the data similar to one or more second images for a same diagnostic context; and
a graphics processing unit configured to render the first image from the data using the settings output by the application of the data to the machine-learnt model, the first image being a two-dimensional representation.

23. The system of claim 22 wherein the machine-learnt model is machine learnt with deep learning.

24. The system of claim 22 wherein the rendering parameters comprise material properties, viewing properties, lighting properties, windowing properties, and internal renderer properties, and wherein the graphics processing unit is configured to render with path tracing using the settings.

Assignees

• Siemens Healthcare Gmbh

Inventors

• Petkov Kaloian
• Miao Shun
• Yu Daphne
• Georgescu Bogdan
• Engel Klaus
• Mansi Tommaso
• Comaniciu Dorin

Classifications

• G06V10/82

  using neural networks · CPC title

• G06V10/764

  using classification, e.g. of video objects · CPC title

• G06N7/01

  Probabilistic graphical models, e.g. probabilistic networks · CPC title

• G06F18/217

  Validation; Performance evaluation; Active pattern learning techniques · CPC title

• G06F18/2414

  Smoothing the distance, e.g. radial basis function networks [RBFN] · CPC title