Interactive mixed reality system and uses thereof

Patent metadata

Field	Value
Publication number	US-9251721-B2
Application number	US-201113514474-A
Country	US
Kind code	B2
Filing date	Apr 8, 2011
Priority date	Apr 9, 2010
Publication date	Feb 2, 2016
Grant date	Feb 2, 2016

Abstract

An interactive mixed reality simulator is provided that includes a virtual 3D model of internal or hidden features of an object; a physical model or object being interacted with; and a tracked instrument used to interact with the physical object. The tracked instrument can be used to simulate or visualize interactions with internal features of the physical object represented by the physical model. In certain embodiments, one or more of the internal features can be present in the physical model. In another embodiment, some internal features do not have a physical presence within the physical model.

First claim

We claim:

1. An interactive mixed reality system, comprising:
a physical model of an object;
a tracked instrument for interacting with and penetrating the physical model, wherein the physical model is penetrable through an external or internal surface at any location and at any orientation of the tracked instrument, wherein the tracked instrument is moveable freely within the physical model;
a tracking system for tracking the position and the orientation of the tracked instrument within and near the physical model;
a virtual model, stored in a memory, comprising one or more 3-D virtual representations of features of the object, the features comprising internal or hidden features of the object, wherein the one or more 3-D virtual representations are abstract or concrete representations;
a 3-D perspective controller device for selecting a perspective of the virtual model, wherein the perspective is selectable by a user independently of the position and orientation of the tracked instrument; and
a simulation module that
accesses the virtual model stored in the memory;
receives input from the tracking system regarding the position and the orientation of the tracked instrument within or near the physical model;
determines, based on the input from the tracking system, a corresponding interaction between the tracked instrument within or near the physical model and the one or more 3-D virtual representations of the virtual model;
determines a consequence of the corresponding interaction; and
renders, to a display, from the perspective of the virtual model selected on the 3-D perspective controller, a particular set of the one or more 3-D virtual representations, and the consequence of the corresponding interaction.

2. The interactive mixed reality system according to claim 1 , wherein the object is a living entity.

3. The interactive mixed reality system according to claim 1 , wherein the object is a non-living object.

4. The interactive mixed reality system according to claim 1 , wherein the virtual model comprises anatomically correct models of internal organs, structures and systems.

5. The interactive mixed reality system according to claim 1 , wherein in response to a repetitive training mode, the simulation module determines and renders a different set from the particular set of the one or more 3-D virtual representations, wherein the different set provides anatomical variability and levels of procedural difficulty.

6. The interactive mixed reality system according to claim 5 , wherein the different set of the one or more 3-D virtual representations comprises changes to locations and sizes of structures in the particular set of the one or more 3-D virtual representations.

7. The interactive mixed reality system according to claim 1 , wherein at least one of the features of the object is not physically present on the physical model, wherein the at least one of the features of the object that is not physically present on the physical model is provided by the virtual model.

8. The interactive mixed reality system according to claim 1 , wherein the simulation module comprises computer implemented instructions of a simulation of a procedure.

9. The interactive mixed reality system according to claim 1 , wherein the simulation module comprises computer implemented instructions of a visualization of a procedure.

10. The interactive mixed reality system according to claim 9 , wherein the computer implemented instructions of the visualization of the procedure comprise instructions to provide a virtual representation of internal structures deforming as the tracked instrument approaches a deformable structure.

11. The interactive mixed reality system according to claim 1 , wherein the rendering to the display by the simulation module further comprises rendering the tracked instrument as a virtual representation in the virtual model.

12. The interactive mixed reality system according to claim 11 , wherein the virtual representation of the tracked instrument is shown interacting with features of the object in the virtual model that are not physically present in the physical model.

13. The interactive mixed reality system according to claim 1 , wherein the tracked instrument is registered within the virtual model such that discrete adjustments, manipulations, and interventions of the tracked instrument are reflected in the virtual model.

14. The interactive mixed reality system according to claim 1 , wherein the tracked instrument expresses tactile, haptic or visual cues for feedback to a user.

15. The interactive mixed reality system according to claim 14 , wherein the tracked instrument is a syringe and the visual cues comprise:
one or more light emitting diodes to indicate that a vein or artery in the virtual model has been entered by a needle connected to the syringe.

16. The interactive mixed reality system according to claim 1 , wherein the tracked instrument comprises a deformable tool, wherein the tracking system comprises a magnetic sensor near a tip end of the deformable tool, the tool being insertable into the physical model.

17. The interactive mixed reality system according to claim 1 , further comprising a second tracked instrument.

18. The interactive mixed reality system according to claim 17 , wherein the second tracked instrument comprises an ultrasound probe.

19. The interactive mixed reality system according to claim 18 , wherein based on a position and orientation of the ultrasound probe with respect to the physical and virtual models, the simulation module further generates a simulated ultrasound image.

20. The interactive mixed reality system of claim 19 , wherein generating the ultrasound image comprises generating virtual representations of internal structures deforming according to how hard the tracked instrument is impinging the external surface of the physical model.

21. A system for interacting with features of an object, the system comprising:
a physical model of an object, wherein the physical model is penetrable through an external or internal surface at any location and at any orientation;
one or more tracked instrument for interacting with the physical model;
a tracking module for tracking a position and an orientation of the one or more tracked instrument relative to the physical model;
a virtual model, stored in a memory, comprising one or more 3-D virtual representations of features of the object, the features comprising internal or hidden features of the object, the one or more 3-D virtual representations comprising abstract or concrete representations;
a 3-D perspective controller device for selecting a perspective of the virtual model, wherein the perspective is selectable by a user independently of the position and orientation of the one or more tracked instrument; and
a simulation module that:
accesses the virtual model stored in the memory;
receives input from the tracking module regarding the position and orientation of the one or more tracked instrument relative to the physical model;
determines, based on the input from the tracking system, a corresponding interaction between the one or more tracked instrument and the one or more 3-D virtual representations of the virtual model;
determines a consequence of the corresponding interaction; and
renders, to a display, a virtual representation comprising a particular set of the one or more 3-D virtual representations, the corresponding interaction, and the consequence of the corresponding interaction; an

Assignees

• Lampotang Samsun
• Gravenstein Nikolaus
• Lizdas David Erik
• Luria Isaac Thomas
• Peterson Matthew James
• Univ Florida

Inventors

• Lampotang Samsun
• Gravenstein Nikolaus
• Lizdas David Erik
• Luria Isaac Thomas
• Peterson Matthew James

Classifications

• G09B23/285 Primary

  for injections, endoscopy, bronchoscopy, sigmoidscopy, insertion of contraceptive devices or enemas · CPC title

• G09B23/286 Primary

  for scanning or photography techniques, e.g. X-rays, ultrasonics · CPC title

• G06T19/006 Primary

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• B33Y80/00

  Products made by additive manufacturing · CPC title

• G09B9/00

  Simulators for teaching or training purposes (for the use of weapons F41; computing aspects G06; {protocols for games, networked simulations or virtual reality H04L67/131}) · CPC title