Real-time and accurate soft tissue deformation prediction

Patent metadata

Field	Value
Publication number	US-10636142-B2
Application number	US-201815958019-A
Country	US
Kind code	B2
Filing date	Apr 20, 2018
Priority date	Apr 20, 2018
Publication date	Apr 28, 2020
Grant date	Apr 28, 2020

Abstract

For soft tissue deformation prediction, a biomechanical or other tissue-related physics model is used to find an instantaneous state of the soft tissue. A machine-learned artificial neural network is applied to predict the position of volumetric elements (e.g., mesh node) from the instantaneous state. Since the machine-learned artificial neural network may predict quickly (e.g., in a second or less), the soft tissue position at different times or a further time given the instantaneous state is provided in real-time without the minutes of physics model computation. For example, a real-time, biomechanical solver is provided, allowing interaction with the soft tissue model, while still getting accurate results. The accuracy allows for generating images of a soft tissue with greater accuracy and/or the benefit of user interaction in real-time.

First claim

We claim:

1. A method for soft tissue deformation prediction, the method comprising:
generating a training database by simulating soft tissue deformation with a biomechanics solver for a plurality of samples over time with first temporal increments;
machine training an artificial neural network to predict a deformation characteristic of a nodal position of a soft tissue model as a function of time from locations and instantaneous biomechanical characteristics of the samples over time calculated from the biomechanical solver;
predicting, by the trained artificial neural network, the soft tissue position from a current location and instantaneous biomechanical characteristics from the biomechanical solver, the predicted soft tissue position being with a second temporal increment greater than the first temporal increment; and
generating an image of the soft tissue after deformation.

2. The method of claim 1 wherein simulating and predicting comprise simulating and predicting with the second temporal increments being at least 500 times the first temporal increments.

3. The method of claim 1 wherein machine training comprises performing learning a deep neural network, with an architecture based on feed-forward, and/or recurrent neural networks with long-short-term memory or gated recurrent units.

4. The method of claim 1 wherein machine training and predicting comprise machine training and predicting with the instantaneous forces comprising a sum of external forces, surface forces, and internal forces.

5. The method of claim 1 wherein predicting comprises determining the soft tissue position from a temporal integration of an acceleration.

6. The method of claim 5 wherein determining comprises temporally integrating a velocity from the acceleration and temporally integrating the soft tissue position from the velocity.

7. The method of claim 6 further comprising applying boundary conditions for the acceleration, the velocity, and/or the soft tissue position after predicting.

8. The method of claim 1 wherein generating comprises simulating the soft tissue deformation as a sequence of meshes over time with the first temporal increments, wherein the locations for machine training are locations of nodes of the meshes, and wherein predicting the soft tissue position is predicting a nodal position of a mesh of the soft tissue.

9. The method of claim 1 wherein generating the image comprises reconstructing an image from scan data where the reconstructing uses the predicted soft tissue deformations, generating the image with a registration from different scans of llthell a patient where the registration is based on the soft tissue deformations, or generating an image simulated from a modeled change where the soft tissue position is used to solve a model used to simulate the modeled change.

10. The method of claim 1 wherein predicting comprises predicting in response to a user input of a change to the soft tissue state.

11. A method for soft tissue deformation prediction by a medical imaging system, the method comprising:
scanning, by the medical imaging system, a patient;
estimating a mesh for a soft tissue from scan data from the scanning, the mesh representing the soft tissue at a first time;
fitting a biomechanical model;
predicting, by a machine-learned artificial neural network, a deformation of the mesh at a second time from a mechanical state indicated by the biomechanical model for the first time; and
generating an image of the soft tissue, the image being based on the predicted deformation.

12. The method of claim 11 wherein predicting comprises predicting the deformation as a position of the mesh at the second time.

13. The method of claim 11 wherein the machine-learned artificial neural network was trained using a first time increment and wherein predicting comprises predicting by the machine-learned artificial neural network with a difference between the first and second times being greater than the first time increment.

14. The method of claim 11 wherein predicting comprises predicting with the mechanical state comprising a sum of instantaneous forces.

15. The method of claim 11 wherein predicting comprises predicting an acceleration and determining a nodal location of a mesh from a temporal integration scheme of the acceleration.

16. The method of claim 11 further comprising receiving user input, and wherein predicting comprises predicting based on the user input and within a second of receiving the user input.

17. A system for soft tissue deformation prediction, the system comprising:
at least one medical imaging system configured to acquire scan data representing a patient;
an image processor configured to model physics in of soft tissue of the patient from the scan data, and to predict a location of the soft tissue by application of a current mechanical state from the model of physics to a machine-trained artificial neural network;
a display configured to display an image based on the soft tissue model location.

18. The system of claim 17 wherein the machine-trained artificial neural network was trained with simulations at a first time step, and wherein the image processor is configured to predict the soft tissue model with a second time step larger than the first time step.

19. The system of claim 17 wherein the image processor is configured to model the physics with a biomechanical model, and wherein the application to the machine-trained artificial neural network results in output of an acceleration, the soft tissue model location being from a temporal integration from the predicted acceleration.

20. The system of claim 17 further comprising a user input, wherein the image processor is configured to predict the soft tissue model location in real-time with input from the user input.

Assignees

• Siemens Healthcare Gmbh

Inventors

• Mansi Tommaso
• Meister Felix
• Passerini Tiziano
• Mihalef Viorel

Classifications

• G16H50/70

  for mining of medical data, e.g. analysing previous cases of other patients · CPC title

• A61B5/7267 Primary

  involving training the classification device · CPC title

• A61B5/1079

  using optical or photographic means · CPC title

• G06T2207/20081

  Training; Learning · CPC title

• G06T7/0012 Primary

  Biomedical image inspection · CPC title