Content-based medical image rendering based on machine learning

Patent metadata

Field	Value
Publication number	US-9760690-B1
Application number	US-201615191043-A
Country	US
Kind code	B1
Filing date	Jun 23, 2016
Priority date	Mar 10, 2016
Publication date	Sep 12, 2017
Grant date	Sep 12, 2017

Abstract

An artificial intelligence agent is machine trained and used to provide physically-based rendering settings. By using deep learning and/or other machine training, settings of multiple rendering parameters may be provided for consistent imaging even in physically-based rendering.

First claim

We claim:

1. A method for content-based rendering based on machine learning in a rendering system, the method comprising:
loading, from memory, a medical dataset representing a three-dimensional region of a patient;
applying, by a machine, the medical dataset to a machine-learnt model, the machine-learned model trained with deep learning to extract features from the medical dataset and trained to output values for two or more physically-based rendering parameters based on input of the medical dataset, the two or more physically-based rendering parameters being settings of a physically-based renderer, the settings used by the physically-based renderer to control rendering from three dimensions to two-dimensions, at least one of the two or more rendering parameters being controls for data consistency handling, lighting design, viewing design, material propriety, or internal renderer property;
rendering, by the physically-based renderer, a photorealistic image of the three-dimensional region of the patient using the output values resulting from the applying as the settings to control the rendering from the medical dataset, the rendering of the medical dataset of the three-dimensional region being to the photorealistic image in the two-dimensions; and
transmitting the photorealistic image.

2. The method of claim 1 further comprising loading patient information other than the medical dataset representing the three-dimensional region of the patient.

3. The method of claim 1 wherein applying comprises applying to output the values for the two or more physically-based rendering parameters as all of the physically-based rendering parameters other than viewing design based on applying user input of the viewing design and the medical dataset.

4. The method of claim 1 wherein applying comprises applying with the machine-learnt model trained to output the values resulting in the photorealistic image corresponding to a standard image despite differences in the medical dataset.

5. The method of claim 1 wherein applying comprises applying with the machine-learnt model comprising a regression, classification, or reinforcement learnt model.

6. The method of claim 1 wherein applying comprises applying with the deep learning as a deep neural network.

7. The method of claim 1 wherein applying comprises applying with the machine-learnt model as a deep reinforcement learnt model.

8. The method of claim 1 wherein rendering comprises rendering with unbiased path tracing.

9. The method of claim 1 wherein transmitting comprises transmitting as part of a diagnostic report, as an initial image of an interactive viewing, or as an overlay in augmented reality.

10. The method of claim 1 further comprising:
measuring ambient light with a light sensor;
wherein applying comprises applying the ambient light and the medical dataset, the machine-learnt model trained to output the values based in part on the ambient light.

11. A method for machine training for content-based rendering in a machine training system, the method comprising:
inputting first volume data of a volume of a patient, a first image of the volume, and first values of path tracing rendering parameters to training of an artificial intelligence, the path tracing rendering parameters being settings to control rendering from the volume to a two-dimensional image;
machine training, with a machine, the artificial intelligence to output second values of the path tracing rendering parameters for a second volume data where the second values control the rendering from the volume to provide a second rendered image of the second volume modeled on the first image; and
storing the trained artificial intelligence.

12. The method of claim 11 wherein inputting comprises inputting the patient non-image information to the training.

13. The method of claim 11 wherein training comprises training the artificial intelligence to output the second values as two or more of data consistency, transfer function, lighting, and viewing parameters.

14. The method of claim 11 wherein training comprises training the artificial intelligence to output the second values based on user selected viewing camera parameters, the path tracing rendering parameters for which second values are to be output being other than the viewing camera parameters.

15. The method of claim 11 wherein training so the second rendered image is modeled after the first rendered image comprises training with a metric of similarity.

16. The method of claim 11 wherein the first image comprises a reference photograph or video of the patient.

17. The method of claim 11 wherein inputting comprises perturbing the path tracing rendering parameters, creating a collection of sets of the path tracing rendering parameters, and wherein training comprises training based on selection of a sub-set of the sets.

18. The method of claim 17 wherein selection of the sub-set comprises selection by a user based on images rendered using the sets.

19. The method of claim 11 wherein machine training comprises deep learning with regression, classification, or reinforcement learning.

20. The method of claim 19 wherein machine training comprises deep reinforcement learning with a similarity of the second rendered image to the first image as a reinforcement.

21. The method of claim 20 wherein deep reinforcement learning comprises selecting with a probability distribution of different similarities including the similarity.

22. A system for content-based rendering based on machine learning, the system comprising:
a medical scanner configured to scan a patient;
a machine configured to output settings for rendering parameters by application of data from the scan to a machine-learnt model, the rendering parameters being controls for performing rendering from a volume to a two-dimensional image, the rendering parameters including windowing, transfer function, and lighting, and the settings learned to provide a first image from the data similar to one or more second images for a same diagnostic context; and
a graphics processing unit configured to render the first image from the data using the settings output by the application of the data to the machine-learnt model, the first image being a two-dimensional representation.

23. The system of claim 22 wherein the machine-learnt model is machine learnt with deep learning.

24. The system of claim 22 wherein the rendering parameters further comprise material properties, viewing properties, lighting properties and internal renderer properties, and wherein the graphics processing unit is configured to render with path tracing using the settings.

Assignees

• Siemens Healthcare Gmbh

Inventors

• Petkov Kaloian
• Miao Shun
• Yu Daphne
• Georgescu Bogdan
• Engel Klaus
• Mansi Tommaso
• Comaniciu Dorin

Classifications

• G06V10/82

  using neural networks · CPC title

• G06V10/764

  using classification, e.g. of video objects · CPC title

• G06F18/217

  Validation; Performance evaluation; Active pattern learning techniques · CPC title

• G06N7/01

  Probabilistic graphical models, e.g. probabilistic networks · CPC title

• G06F18/2414

  Smoothing the distance, e.g. radial basis function networks [RBFN] · CPC title