Surgeon head-mounted display apparatuses

Patent metadata

Field	Value
Publication number	US-11734901-B2
Application number	US-202117533169-A
Country	US
Kind code	B2
Filing date	Nov 23, 2021
Priority date	Feb 3, 2015
Publication date	Aug 22, 2023
Grant date	Aug 22, 2023

Abstract

An augmented reality surgical system includes a head mounted display (HMD) with a see-through display screen, a motion sensor, a camera, and computer equipment. The motion sensor outputs a head motion signal indicating measured movement of the HMD. The computer equipment computes the relative location and orientation of reference markers connected to the HMD and to the patient based on processing a video signal from the camera. The computer equipment generates a three dimensional anatomical model using patient data created by medical imaging equipment, and rotates and scales at least a portion of the three dimensional anatomical model based on the relative location and orientation of the reference markers, and further rotate at least a portion of the three dimensional anatomical model based on the head motion signal to track measured movement of the HMD. The rotated and scaled three dimensional anatomical model is displayed on the display screen.

First claim

What is claimed is:

1. A method for performing a surgical procedure, said method comprising:
providing an augmented reality surgical system including:
a display comprising a see-through display screen that display images while allowing transmission of ambient light therethrough;
a motion sensor connected to the display and configured to output a motion signal indicating measured movement of the display; and
at least one camera configured to observe a first set of reference markers connected to a patient, and a second set of reference markers connected to a surgical tool located within a surgical room; and
a computer,

computing the relative location and orientation of the display and the first and second set of reference markers;
generating a three dimensional anatomical image using patient data created by medical imaging equipment that has imaged a portion of the patient;
generating a video signal based on at least a portion of the three dimensional anatomical image and the location and orientation of the first and second set of reference markers coupled to the patient and the surgical instrument; and
output a video signal to the display screen on the display,
wherein the computer is configured to animate movement of a displayed graphical representation of the patient, surgical tool or surgical site to illustrate a planned procedure relative to a defined location of the surgical site or other targeted anatomical site.

2. The method of claim 1 , further comprising displaying a graphical representation on the display screen of the at least a portion of a three dimensional anatomical model oriented and scaled to provide a displayed overlay on the portion of the patient that was imaged by the medical imaging equipment.

3. The method of claim 1 , further comprising displaying a graphical representation of a medical procedure on the display overlaid on the portion of the patient that was imaged by the medical imaging equipment.

4. A method for performing a surgical procedure, said method comprising:
providing an augmented reality surgical system including:
a positioning tracking system; and
a computer system

tracking a location of a surgical tool, a head mounted display, and a surgical site via the positioning tracking system;
generating an image model of a targeted site of the patient by the computer system using patient data from an imaging device, wherein the image model includes reference markers that assist in correlating between virtual locations and physical locations of the patient;
processing, by the computer system, the present locations the head mounted display, the surgical site and the surgical tool obtained by the positioning tracking system; and
generating a graphical representation of the patient to the head mounted display for a user to view, wherein the graphical representation is generated using the reference markers contained in the image model to generate,
wherein the computer system is configured to animate movement of the displayed graphical representation of the patient, surgical tool or surgical site to illustrate a planned procedure relative to a defined location of the surgical site or other targeted anatomical site.

5. The method of claim 4 , further comprising displaying graphical representations of the surgical tool or surgical site on the display screen of the head mounted display.

6. The method of claim 4 , wherein imaging device may include one of an x-ray device, endoscope cameras, magnetic resonance imaging equipment, computed tomography scanning equipment, three-dimensional ultrasound equipment, and endoscopic equipment.

7. The method of claim 4 , wherein the computer may receive non-image patient data and display the non-image data into a graphical representation to be displayed on the head mounted display.

8. The method of claim 4 , wherein the positioning tracking system includes a plurality of cameras spaced apart at defined locations within an operating room and each having a field of view that can observe objects to be tracked.

9. The method of claim 4 , wherein the positioning tracking system may use electromagnetic ranging and trackers, ultrasound emitters/sensors for ranging and trackers.

10. A method for displaying multiple video streams to a user during a surgical procedure, said method comprising:
providing an augmented reality surgical system including:
a display comprising a see-through display screen that display images while allowing transmission of ambient light therethrough;
a motion sensor connected to the display and configured to output a motion signal indicating measured movement of the display; and
a computer configured to receive a plurality patient data including a three dimensional anatomical image of the patient and relative location and orientation of the patient and to control which or all of the plurality of patient data to output as a video signal to the display screen,
outputting the video signal to the display screen,

wherein the computer is configured to animate movement of a displayed graphical representation of the patient, surgical tool or surgical site to illustrate a planned procedure relative to a defined location of the surgical site or other targeted anatomical site.

11. The method of claim 10 , wherein: the computer is communicatively connected to a data server to receive the patient data, one of the which is a data generated by medical imaging equipment, another one of the patient data is information from a patient database defining a patient's medical history, and another one of the patient data comprises data generated by medical equipment based on real-time monitoring of the patient's vitals.

12. The method of claim 10 , wherein: the computer is configured to convert the patient data into a graphical display and to output the selected one or all of the plurality of patient data into video streams as a video signal to the display screen.

13. The method of claim 10 , wherein the computer communicates video from the video camera through a network to another display worn by another person.

14. The method of claim 10 , wherein the computer provides a graphical representation of a virtual trajectory of a surgical tool to the patient's surgical site.

15. The method of claim 10 , wherein the surgical tool is a drill bit.

Assignees

• Globus Medical Inc

Inventors

• JONES Kenneth Milton
• Popoolapade John
• CALLOWAY Thomas
• Lemoine Thierry
• Jutteau Christian
• Bruzy Christophe
• James Yannick
• LAGUARDA Joachim
• XING Dong-Mei Pei
• Gorges Sebastien
• Yarin Paul Michael

Classifications

• G06T19/006 Primary

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• A61B34/10

  Computer-aided planning, simulation or modelling of surgical operations · CPC title

• A61B90/37

  Surgical systems with images on a monitor during operation · CPC title

• G06F3/011

  Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00) · CPC title

• G06F3/012

  Head tracking input arrangements · CPC title