Interactive mixed reality system and uses thereof

Patent metadata

Field	Value
Publication number	US-9626805-B2
Application number	US-201514980891-A
Country	US
Kind code	B2
Filing date	Dec 28, 2015
Priority date	Apr 9, 2010
Publication date	Apr 18, 2017
Grant date	Apr 18, 2017

Abstract

An interactive mixed reality simulator is provided that includes a virtual 3D model of internal or hidden features of an object; a physical model or object being interacted with; and a tracked instrument used to interact with the physical object. The tracked instrument can be used to simulate or visualize interactions with internal features of the physical object represented by the physical model. In certain embodiments, one or more of the internal features can be present in the physical model. In another embodiment, some internal features do not have a physical presence within the physical model.

First claim

We claim:

1. An interactive mixed reality system, comprising:
a physical model of an object, wherein the physical model is penetrable through an external or internal surface at any location and at any orientation;
one or more tracked instruments for interacting with the physical model;
a tracking system for tracking a position and an orientation of the one or more tracked instruments relative to the physical model;
a virtual model, stored in a memory, comprising one or more 3-D virtual representations of features of the object, the features comprising internal or hidden features of the object, the one or more 3-D virtual representations comprising abstract or concrete representations;
a simulation module that:
accesses the virtual model stored in the memory;
receives input from the tracking module regarding the position and orientation of the one or more tracked instruments relative to the physical model;
determines, based on the input from the tracking system, a corresponding interaction between the one or more tracked instruments and the one or more 3-D virtual representations of the virtual model;
determines a consequence of the corresponding interaction; and
renders, to a display, a virtual representation comprising a particular set of the one or more 3-D virtual representations, the corresponding interaction, and the consequence of the corresponding interaction,
wherein the corresponding interaction is a physical interaction between the one or more tracked instruments within the physical model and the internal or hidden features of the object in the virtual model.

2. The interactive mixed reality system according to claim 1 , wherein the rendering is from a perspective of the virtual model that is selectable by a user independently of the position and orientation of at least one tracked instrument.

3. The interactive mixed reality system according to claim 1 , wherein the rendering is from a perspective of the virtual model responsive to a zoom, magnification, or granularity determination of the system.

4. The interactive mixed reality system according to claim 1 , wherein the virtual model comprises anatomically correct models of internal organs, structures and systems.

5. The interactive mixed reality system according to claim 1 , wherein in response to a repetitive training mode, the simulation module determines and renders a different set from the particular set of the one or more 3-D virtual representations, wherein the different set provides anatomical variability and levels of procedural difficulty.

6. The interactive mixed reality system according to claim 5 , wherein the different set of the one or more 3-D virtual representations comprises changes to locations and sizes of structures in the particular set of the one or more 3-D virtual representations.

7. The interactive mixed reality system according to claim 1 , wherein at least one of the features of the object is not physically present on the physical model and is provided by the virtual model.

8. The interactive mixed reality system according to claim 1 , wherein the simulation module comprises computer implemented instructions of a simulation or a visualization of a procedure.

9. The interactive mixed reality system according to claim 8 , wherein the computer implemented instructions of the visualization of the procedure comprise instructions to provide a virtual representation of internal structures deforming as the tracked instrument approaches a deformable structure.

10. The interactive mixed reality system according to claim 1 , wherein the rendering to the display by the simulation module further comprises rendering at least one of the one or more tracked instruments as a virtual representation in the virtual model.

11. The interactive mixed reality system according to claim 10 , wherein the virtual representation of at least one of the one or more tracked instruments is shown interacting with features of the object in the virtual model that are not physically present in the physical model.

12. The interactive mixed reality system according to claim 1 , wherein at least one of the one or more tracked instruments is registered within the virtual model such that discrete adjustments, manipulations, and interventions of at least one of the one or more tracked instruments are reflected in the virtual model.

13. The interactive mixed reality system according to claim 1 , wherein at least one of the one or more tracked instruments expresses tactile, haptic or visual cues for feedback to a user.

14. The interactive mixed reality system according to claim 13 , wherein the at least one of the one or more tracked instruments is a syringe and the visual cues comprise:
one or more light emitting diodes to indicate that a vein or artery in the virtual model has been entered by a needle connected to the syringe.

15. The interactive mixed reality system according to claim 1 , wherein a virtual representation of the virtual model further comprises an ultrasound, X-ray, fluoroscopy, MRI, CT, or other medical image of features of the physical and virtual models on which the interaction of at least one of the one or more tracked instruments and the physical and virtual models is indicated.

16. The interactive mixed reality system according to claim 1 , wherein at least one of the one or more tracked instruments comprises a deformable tool, wherein the tracking system comprises a magnetic sensor near a tip end of the deformable tool, the tool being insertable into the physical model.

17. The interactive mixed reality system according to claim 1 , wherein at least one of the one or more tracked instruments comprises an ultrasound probe.

18. The interactive mixed reality system according to claim 17 , wherein based on a position and orientation of the ultrasound probe with respect to the physical and virtual models, the simulation module further generates a simulated ultrasound image.

19. The interactive mixed reality system according to claim 18 , wherein generating the ultrasound image comprises generating virtual representations of internal structures deforming according to how hard the one or more tracked instruments are impinging the external surface of the physical model.

20. The interactive mixed reality system according to claim 18 , wherein generating the ultrasound image comprises generating virtual representations of internal structures deforming according to the proximity of the one or more tracked instruments to an internal structure of the virtual model or the physical model.

21. The interactive mixed reality system according to claim 1 , further comprising at least two tracked instruments, wherein at least one of the two tracked instruments comprises an ultrasound probe.

22. The interactive mixed reality system according to claim 1 , wherein at least one of the one or more tracked instruments penetrates through the external or internal surface of the physical model and is moveable freely within and near the physical model, and
wherein the tracking system tracks the position and the orientation of a tracked instrument when the tracked instrument is unsighted within the physical model.

23. The interactive mixed reality system according to claim 1 , wherein at least one tracked instrument is held in at least one of a user's hands and directly manipulated without an intermediary device or a mediated interface.

24. The interactive mixed reality system according to clai

Assignees

• Univ Florida
• Univ Florida

Inventors

• Lampotang Samsun
• Gravenstein Nikolaus
• Lizdas David Erik
• Luria Isaac Thomas

Classifications

• G09B23/285 Primary

  for injections, endoscopy, bronchoscopy, sigmoidscopy, insertion of contraceptive devices or enemas · CPC title

• G09B23/286

  for scanning or photography techniques, e.g. X-rays, ultrasonics · CPC title

• G06T19/006 Primary

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G06F3/017

  Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883) · CPC title

• G09B23/28

  for medicine · CPC title