Surgeon head-mounted display apparatuses

Patent metadata

Field	Value
Publication number	US-11062522-B2
Application number	US-201916706948-A
Country	US
Kind code	B2
Filing date	Dec 9, 2019
Priority date	Feb 3, 2015
Publication date	Jul 13, 2021
Grant date	Jul 13, 2021

Abstract

An augmented reality surgical system includes a head mounted display (HMD) with a see-through display screen, a motion sensor, a camera, and computer equipment. The motion sensor outputs a head motion signal indicating measured movement of the HMD. The computer equipment computes the relative location and orientation of reference markers connected to the HMD and to the patient based on processing a video signal from the camera. The computer equipment generates a three dimensional anatomical model using patient data created by medical imaging equipment, and rotates and scales at least a portion of the three dimensional anatomical model based on the relative location and orientation of the reference markers, and further rotate at least a portion of the three dimensional anatomical model based on the head motion signal to track measured movement of the HMD. The rotated and scaled three dimensional anatomical model is displayed on the display screen.

First claim

What is claimed is:

1. An augmented reality surgical system comprising:
a display comprising a see-through display screen that display images while allowing transmission of ambient light therethrough;
a motion sensor connected to the display and configured to output a motion signal indicating measured movement of the display; and
at least one camera coupled to a portion of the display and configured to observe reference markers connected to a patient, and reference markers connected to a surgical tool located within a surgical room;
computer equipment configured to:
compute the relative location and orientation of the display and the reference markers connected to the patient based on processing an image signal from the at least one camera;
generate a three dimensional anatomical image using patient data created by medical imaging equipment that has imaged a portion of the patient,
rotate and scale at least a portion of the three dimensional anatomical model based on the relative location and orientation of the reference markers connected to the head mounted display and the reference markers connected to the patient; and generate a video signal based on at least a portion of the three dimensional anatomical image and the location and orientation of the reference makers coupled to the patient and a surgical instrument, and
output a video signal to the display screen on the display,
wherein the video signal is a graphical representation of a virtual trajectory of the surgical instrument that intersects a target location on a patient, the target location corresponds to a point the surgical instrument impacts the three dimensional anatomical image which is a oriented and scaled to match the patient's anatomy, and
wherein the virtual trajectory is a continuously updated and dynamically displayed to a surgeon who is repositioning and reorienting the surgical instrument.

2. The augmented reality surgical system of claim 1 , wherein the user sees a graphical representation on the display screen of the at least a portion of the three dimensional anatomical model oriented and scaled to provide a displayed above the patient that was imaged by the medical imaging equipment.

3. The augmented reality surgical system of claim 1 , wherein the computer equipment compares patterns of anatomical objects in a video stream from at least one video camera coupled to the surgical system to patterns of anatomical objects in the patient data created by the medical imaging equipment, and controls generation of the three dimensional anatomical model from the patient data responsive to identifying a threshold level of correspondence between the compared patterns of anatomical objects.

4. The augmented reality surgical system of claim 1 , wherein the computer equipment compares patterns of anatomical objects in a video stream from at least one video camera to patterns of anatomical objects in the patient data created by the medical imaging equipment, and displays a graphical indicia on the display screen aligned with one of the anatomical objects displayed on the display screen from the rotated and scaled three dimensional anatomical model responsive to identifying a threshold level of correspondence between a pattern of the one of the anatomical objects and a pattern of one of the anatomical objects in the video stream from the at least one video camera.

5. The augmented reality surgical system of claim 1 , wherein the at least one camera is further configured to observe reference markers connected to a surgical tool located within a surgical room; and generate the video signal to include a graphical representation of the surgical tool illustrated at a position relative to the three dimensional anatomical model that is determined based on the relative location and orientation of the reference markers connected to the head mounted display, the reference markers connected to the patient, and the reference markers connected to the surgical tool.

6. The augmented reality surgical system of claim 5 , wherein the computer equipment is further configured to: generate a graphical representation of a virtual trajectory extending from the surgical tool into the three dimensional anatomical model based on the relative location and orientation of the reference markers connected to the head mounted display, the reference markers connected to the patient, and the reference markers connected to the surgical tool; and generate the video signal to include the graphical representation of the virtual trajectory.

7. The augmented reality surgical system of claim 6 , wherein: the computer equipment is further configured to select an image slice from among a plurality of image slices contained in the three dimensional anatomical model, the image slice being selected based on the computer equipment determining that the image slice is traversed by the virtual trajectory extending from the surgical tool, and to generate the video signal to include the image slice.

8. The augmented reality surgical system of claim 7 , wherein a graphical representation of the image slice and the graphical representation of the virtual trajectory that are displayed on the display screen are responsive to the head motion signal from the motion sensor and/or responsive to a command received from a user.

9. The augmented reality surgical system of claim 7 , wherein a graphical representation of the image slice and the graphical representation of the virtual trajectory that are displayed on the display screen to provide a view of the image slice that is perpendicular to a direction of the virtual trajectory.

10. The augmented reality surgical system of claim 9 , wherein: the computer equipment is further configured to identify a target location within the image slice, and to display the target location relative to the graphical representation of the virtual trajectory.

11. An augmented reality surgical system for displaying multiple video streams to a user, the augmented reality surgical system comprising:
a display comprising a see-through display screen that display images while allowing transmission of ambient light therethrough;
a motion sensor connected to the display and configured to output a motion signal indicating measured movement of the display; and
a position tracking system configured to track the location of the surgical tool, and/or a prosthetic, the display, a surgical site and a target location of the patient,
wherein the video signal is a graphical representation of a virtual trajectory of the surgical instrument that intersects a target location on a patient, the target location corresponds to a point the surgical instrument impacts the three dimensional anatomical image which is a oriented and scaled to match the patient's anatomy, and
wherein the virtual trajectory is a continuously updated and dynamically displayed to a surgeon who is repositioning and reorienting the surgical instrument.

12. The augmented reality surgical system of claim 11 , wherein: the computer equipment is communicatively connected to a surgical video server to receive the video streams, one of the video streams comprises data generated by medical imaging equipment, another one of the video streams comprises information from a patient database defining a patient's medical history, and another one of the video streams comprises data generated by medical equipment based on real-time Monitoring of the patient's vitals.

13. The augmented reality surgical system of claim 11 , wherein: the computer equipment is configured to select one of the video streams from among the video streams based on the motion signal, and to output the selected one

Assignees

• Globus Medical Inc
• Global Medical Inc

Inventors

• JONES Kenneth Milton
• Popoolapade John
• CALLOWAY Thomas
• Lemoine Thierry
• Jutteau Christian
• Bruzy Christophe
• James Yannick
• LAGUARDA Joachim
• XING Dong-Mei Pei
• Gorges Sebastien
• Yarin Paul Michael

Classifications

• A61B34/20

  Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis · CPC title

• A61B2034/2055

  Optical tracking systems · CPC title

• A61B2090/368

  changing the image on a display according to the operator's position · CPC title

• G06F3/0346

  with detection of the device orientation or free movement in a three-dimensional [3D] space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors · CPC title

• A61B2090/365

  augmented reality, i.e. correlating a live optical image with another image · CPC title