Surgeon head-mounted display apparatuses

Patent metadata

Field	Value
Publication number	US-2018286136-A1
Application number	US-201816000393-A
Country	US
Kind code	A1
Filing date	Jun 5, 2018
Priority date	Feb 3, 2015
Publication date	Oct 4, 2018
Grant date	—

Abstract

An augmented reality surgical system includes a head mounted display (HMD) with a see-through display screen, a motion sensor, a camera, and computer equipment. The motion sensor outputs a head motion signal indicating measured movement of the HMD. The computer equipment computes the relative location and orientation of reference markers connected to the HMD and to the patient based on processing a video signal from the camera. The computer equipment generates a three dimensional anatomical model using patient data created by medical imaging equipment, and rotates and scales at least a portion of the three dimensional anatomical model based on the relative location and orientation of the reference markers, and further rotate at least a portion of the three dimensional anatomical model based on the head motion signal to track measured movement of the HMD. The rotated and scaled three dimensional anatomical model is displayed on the display screen.

First claim

1 . An augmented reality surgical system comprising:
a head mounted display comprising a see-through display screen that display images while allowing transmission of ambient light therethrough; a motion sensor connected to the head mounted display and configured to output a head motion signal indicating measured movement of the head mounted display; and at least one camera configured to observe reference markers connected to the head mounted display, reference markers connected to a patient, and reference markers connected to a surgical tool located within a surgical room; computer equipment configured to:
compute the relative location and orientation of the reference markers connected to the head mounted display and the reference markers connected to the patient based on processing a video signal from the at least one camera;
generate a three dimensional anatomical model using patient data created by medical imaging equipment that has imaged a portion of the patient;
rotate and scale at least a portion of the three dimensional anatomical model based on the relative location and orientation of the reference markers connected to the head mounted display and the reference markers connected to the patient, and further rotate at least a portion of the three dimensional anatomical model based on the head motion signal to track measured movement of the head mounted display; and
generate a video signal based on the rotated and scaled three dimensional anatomical model;
output the video signal to the display screen of the head mounted display.

2 . The augmented reality surgical system of claim 1 , wherein:
the computer equipment is configured to rotate and scale the at least a portion of the three dimensional anatomical model based on the relative location and orientation of the reference markers connected to the head mounted display and the reference markers connected to the patient to transform the at least a portion of the three dimensional anatomical model to a present perspective view of a user wearing the head mounted display so the user sees a graphical representation on the display screen of the at least a portion of the three dimensional anatomical model oriented and scaled to provide a displayed overlay on the portion of the patient that was imaged by the medical imaging equipment.

3 . The augmented reality surgical system of claim 1 , further comprising at least one video camera connected to the head mounted display to be forward-facing away from the user, wherein the computer equipment compares patterns of anatomical objects in a video stream from the at least one video camera to patterns of anatomical objects in the patient data created by the medical imaging equipment, and controls generation of the three dimensional anatomical model from the patient data responsive to identifying a threshold level of correspondence between the compared patterns of anatomical objects.

4 . The augmented reality surgical system of claim 1 , further comprising at least one video camera connected to the head mounted display to be forward-facing away from the user, wherein the computer equipment compares patterns of anatomical objects in a video stream from the at least one video camera to patterns of anatomical objects in the patient data created by the medical imaging equipment, and displays a graphical indicia on the display screen aligned with one of the anatomical objects displayed on the display screen from the rotated and scaled three dimensional anatomical model responsive to identifying a threshold level of correspondence between a pattern of the one of the anatomical objects and a pattern of one of the anatomical objects in the video stream from the at least one video camera.

5 . The augmented reality surgical system of claim 1 , wherein:
the at least one camera is further configured to observe reference markers connected to a surgical tool located within a surgical room; the computer equipment is further configured to:
compute the relative location and orientation of the reference markers connected to the head mounted display, the reference markers connected to the patient, and the reference markers connected to the surgical tool based on processing a video signal from the at least one camera; and
generate the video signal to include a graphical representation of the surgical tool illustrated at a position relative to the three dimensional anatomical model that is determined based on the relative location and orientation of the reference markers connected to the head mounted display, the reference markers connected to the patient, and the reference markers connected to the surgical tool.

6 . The augmented reality surgical system of claim 5 , wherein the computer equipment is further configured to:
generate a graphical representation of a virtual trajectory extending from the surgical tool into the three dimensional anatomical model based on the relative location and orientation of the reference markers connected to the head mounted display, the reference markers connected to the patient, and the reference markers connected to the surgical tool; and generate the video signal to include the graphical representation of the virtual trajectory.

7 . The augmented reality surgical system of claim 6 , wherein:
the computer equipment is further configured to select an image slice from among a plurality of image slices contained in the three dimensional anatomical model, the image slice being selected based on the computer equipment determining that the image slice is traversed by the virtual trajectory extending from the surgical tool, and to generate the video signal to include the image slice.

8 . The augmented reality surgical system of claim 7 , wherein:
the computer equipment is further configured to rotate a graphical representation of the image slice and the graphical representation of the virtual trajectory that are displayed on the display screen responsive to the head motion signal from the motion sensor and/or responsive to a command received from a user.

9 . The augmented reality surgical system of claim 7 , wherein:
the computer equipment is further configured to rotate a graphical representation of the image slice and the graphical representation of the virtual trajectory that are displayed on the display screen to provide a view of the image slice that is perpendicular to a direction of the virtual trajectory.

10 . The augmented reality surgical system of claim 9 , wherein:
the computer equipment is further configured to identify a target location within the image slice, and to display the target location relative to the graphical representation of the virtual trajectory.

11 . An augmented reality surgical system for displaying multiple video streams to a user, the augmented reality surgical system comprising:
a head mounted display comprising a see-through display screen that display images while allowing transmission of ambient light therethrough; a motion sensor connected to the head mounted display and configured to output a head motion signal indicating measured movement of the head mounted display; and a computer equipment configured to receive a plurality of video streams from one or more source devices and to control which of the video streams are output as a video signal to the display screen based on the head motion signal.

12 . The augmented reality surgical system of claim 11 , wherein:
the computer equipment is communicatively connected to a surgical video server to receive the video streams, one of the video streams comprises data generated by medical imaging equipment, another one of the video streams comprises information from

Assignees

• Globus Medical Inc

Inventors

• JONES Kenneth Milton
• Popoolapade John
• CALLOWAY Thomas
• Lemoine Thierry
• Jutteau Christian
• Bruzy Christophe
• James Yannick
• LAGUARDA Joachim
• XING Dong-Mei Pei
• Gorges Sebastien
• Yarin Paul Michael

Classifications

• A61B2090/372

  Details of monitor hardware · CPC title

• G06T19/20

  Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts · CPC title

• A61B90/37

  Surgical systems with images on a monitor during operation · CPC title

• A61B2034/105

  Modelling of the patient, e.g. for ligaments or bones · CPC title

• G06T2210/41

  Medical · CPC title