Surgical procedure capture, modelling, and editing interactive playback

Patent metadata

Field	Value
Publication number	US-9396669-B2
Application number	US-13996008-A
Country	US
Kind code	B2
Filing date	Jun 16, 2008
Priority date	Jun 16, 2008
Publication date	Jul 19, 2016
Grant date	Jul 19, 2016

Abstract

A system for generating surgical procedure training media draws upon the realistic data of an actual surgical procedure for realistic training without the risks. A 3D capturing component records three-dimensional model plus imaging data over time of a portion of a patient's body undergoing a surgical procedure. A spatial detection system detects an orientation of a surgical instrument relative to the patient's body during the surgical procedure. A modeling component creates a four-dimensional model (3D model+time) of the portion of the patient's body. Animation such as contingent events, trainee prompts, a virtual surgical instrument, etc., can be added to the model to expand upon the training potential. A user interface processes and edits training media for playback of the four-dimensional model including defining triggers responsive to a trainee simulated surgical inputs to pace sequencing of playback. An interactive player responds to pacing the playback of the editing training media or to a spatially detected simulated surgical instrument held by the student for direct tissue interaction.

First claim

What is claimed is:

1. A method for generating training media, comprising:
capturing three-dimensional data over time of a portion of a human patient's body undergoing a surgical procedure, the capturing being performed at least in part by stereoscopic video cameras or by a three-dimensional camera;
recording physiological readings over time of the portion of the human patient's body, the physiological readings comprise at least a movement cycle;
capturing visual imagery of an exposed portion of the human patient's body;
creating, using one or more processors, a four-dimensional model of the portion of the human patient's body based on the three-dimensional data, the physiological readings, and the visual imagery of the exposed portion of the human patient's body changing over time; and
modifying at least a portion of the three-dimensional data or visual imagery of the human patient's body to maintain privacy of the human patient's body.

2. The method of claim 1 , further comprising editing the training media for playback of the four-dimensional model by defining an object texture map for a viewed surface of the four-dimensional model to correspond to the captured visual imagery.

3. The method of claim 1 , further comprising editing the training media by deleting a period of time from the four-dimensional model.

4. The method of claim 1 , further comprising capturing an orientation of a surgical instrument relative to the human patient's body during the surgical procedure.

5. The method of claim 4 , further comprising removing the surgical instrument from a spatial position the surgical instrument occupies in the four-dimensional model.

6. The method of claim 4 , further comprising editing the training media by defining a surgical instrument placement event that causes playback to pause.

7. The method of claim 6 , further comprising detecting a position of a simulated surgical instrument positioned by a trainee during playback of the training media.

8. The method of claim 7 , further comprising rendering a virtual surgical instrument positioned in the training media corresponding to the simulated surgical instrument.

9. The method of claim 7 , further comprising determining satisfaction of the simulated surgical instrument placement event to continue playback.

10. The method of claim 7 , further comprising:
defining an erroneous instrument placement event in the four-dimensional model; and
in response to the erroneous instrument placement event, displaying a contingent training animation.

11. The method of claim 1 , further comprising editing the training media by changing an orientation of the four-dimensional model to virtually position a trainee.

12. The method of claim 11 , further comprising detecting a head and eye position of the trainee during playback and orienting the four-dimensional model based on the detected head and eye position of the trainee.

13. The method of claim 11 , further comprising displaying a video differentiated for each eye of the trainee for a three-dimensional depiction.

14. The method of claim 1 , further comprising:
selecting a second movement cycle that affects movement of the portion of the patient's body; and
playing back the four-dimensional model of the portion of the patient's body at an increased or reduced speed while rendering a sequence of movements caused by the movement cycle at a normal time rate, wherein the normal time rate is different than the increased or reduced speed, such that the sequence of movements are out of sync with the increased or reduced speed.

15. The method of claim 1 , further comprising:
detecting haptic forces on a surgical instrument used in the surgical procedure;
correlating an orientation of the surgical instrument to a position in the four-dimensional model; and
creating a haptic feedback model for interaction by a simulated surgical instrument with the four-dimensional model.

16. An apparatus for generating surgical procedure training media, comprising:
one or more processors;
an image capturing component for recording three-dimensional imaging data over time of a portion of a human patient's body undergoing a surgical procedure;
a spatial detection system for capturing an orientation of a surgical instrument relative to the human patient's body during the surgical procedure;
a modeling component for creating, using the one or more processors, a four-dimensional model of the portion of the human patient's body based at least on the three-dimensional imaging data and the captured orientation of the surgical instrument and for defining a decision branch in the four-dimensional model and for modifying at least a portion of the three-dimensional imaging data of the human patient's body to maintain privacy of the human patient's body; and
a storage component for storing the four-dimensional model and at least one alternative four-dimensional model, wherein the alternative four-dimensional model is created from recorded three-dimensional data and captured orientations of a surgical instrument during a different surgical procedure performed on a different patient.

17. The apparatus of claim 16 , further comprising a haptic data capturing component for capturing haptic data from the surgical instrument used in the surgical procedure.

18. The apparatus of claim 16 , further comprising a triggering component for switching to a different decision branch of the four-dimensional model based at least in part upon a sensed simulated user action, the different branch corresponding to the four-dimensional model or the alternative four-dimensional model.

19. One or more computer storage media having instructions encoded thereupon that, when executed by one or more processors, cause a computing device to perform acts comprising:
capturing three-dimensional data over time of a portion of a human patient's body undergoing an actual open surgical procedure, the three-dimensional data being captured from at least a stereoscopic video camera or a three-dimensional camera;
capturing visual imagery of an exposed portion of the human patient's body;
capturing an orientation of a surgical instrument relative to the human patient's body during the actual surgical procedure;
capturing haptic forces on the surgical instrument during the actual surgical procedure;
creating a four-dimensional model of the portion of the human patient's body based on the three-dimensional data, the visual imagery of the exposed portion of the human patient's body, the orientation of the surgical instrument, and the haptic forces on the surgical instrument changing over time; and
recolorinq at least a portion of the three-dimensional data or visual imagery of the human patient's body to maintain privacy of the human patient's body.

20. The computer storage media of claim 19 , further comprising:
correlating an orientation of the surgical instrument to a position in the four-dimensional model;
creating a haptic feedback model based at least in part on the captured haptic forces for interaction by a simulated surgical instrument with the four-dimensional model;
editing training media by defining a surgical instrument placement event for the simulated surgical instrument that causes playback to pause; and
detecting a position of the simulated surgical instrument positioned by a trainee during playback of the training media.

21. The computer storage media of claim 20 , further comprising:
defining an erroneous placement of the simulated surgical

Assignees

• Karkanias Chris Demetrios
• Sinclair Michael J
• Hamilton James R
• Rosenbloom Oren
• Van Hoof Hubert
• Microsoft Technology Licensing Llc

Inventors

• Karkanias Chris Demetrios
• Sinclair Michael J
• Hamilton James R
• Rosenbloom Oren
• Van Hoof Hubert

Classifications

• A61B1/044

  for absorption imaging · CPC title

• A61B2019/2223

  Human Necessities · mapped topic

• A61B1/04

  combined with photographic or television appliances · CPC title

• G06T19/00

  Manipulating three-dimensional [3D] models or images for computer graphics · CPC title

• G06T2210/41

  Medical · CPC title