Surgeon head-mounted display apparatuses

Patent metadata

Field	Value
Publication number	US-10013808-B2
Application number	US-201615013594-A
Country	US
Kind code	B2
Filing date	Feb 2, 2016
Priority date	Feb 3, 2015
Publication date	Jul 3, 2018
Grant date	Jul 3, 2018

Abstract

An augmented reality surgical system includes a head mounted display (HMD) with a see-through display screen, a motion sensor, a camera, and computer equipment. The motion sensor outputs a head motion signal indicating measured movement of the HMD. The computer equipment computes the relative location and orientation of reference markers connected to the HMD and to the patient based on processing a video signal from the camera. The computer equipment generates a three dimensional anatomical model using patient data created by medical imaging equipment, and rotates and scales at least a portion of the three dimensional anatomical model based on the relative location and orientation of the reference markers, and further rotate at least a portion of the three dimensional anatomical model based on the head motion signal to track measured movement of the HMD. The rotated and scaled three dimensional anatomical model is displayed on the display screen.

First claim

The invention claimed is:

1. An augmented reality surgical system for displaying multiple video streams to a user, the augmented reality surgical system comprising:
a head mounted display comprising a see-through display screen that display images while allowing transmission of ambient light therethrough;
a motion sensor connected to the head mounted display and configured to output a head motion signal indicating measured movement of the head mounted display; and
a computer equipment configured to receive a plurality of video streams from one or more source devices and to control which of the video streams are output as a video signal to the display screen based on the head motion signal
wherein the computer equipment selects a primary one of the video streams that is output as a video signal to the display screen of the head mounted display based on a pitch component of the head motion signal being less than a first threshold value
wherein based on the pitch component of the head motion signal being above the first threshold value and below a second threshold value, the computer equipment separately feeds different secondary ones of the video streams as a video signal to the display screen of the head mounted display responsive to a yaw component of the head motion signal satisfying different defined yaw threshold ranges, to display virtual floating panels at defined yaw and pitch coordinates relative to the user's head.

2. The augmented reality surgical system of claim 1 , wherein:
the computer equipment is communicatively connected to a surgical video server to receive the video streams, one of the video streams comprises data generated by medical imaging equipment, another one of the video streams comprises information from a patient database defining a patient's medical history, and another one of the video streams comprises data generated by medical equipment based on real-time monitoring of the patient's vitals.

3. The augmented reality surgical system of claim 1 , further comprising: a headband configured to grasp the user's head;
an arm apparatus comprising a two arm friction joint linkage connected between the headband and the head mounted display, the arm apparatus configured to allow rotation of the head mounted display relative to the headband by the user to position the display screen relative to the user's line-of-sight.

4. The augmented reality surgical system of claim 1 , wherein:
the arm apparatus comprises a pivot joint coupled to the headband configured to allow the display screen to be flipped down to a deployed position within the user's line-of sight and flipped-up to a stored position outside the user's line-of sight.

5. The augmented reality surgical system of claim 1 , wherein:
the computer equipment is configured to select one of the video streams from among the video streams based on the head motion signal, and to output the selected one of the video streams as a video signal to the display screen.

6. The augmented reality surgical system of claim 1 , wherein the motion sensor is an orientation sensor configured to output the head motion signal containing a pitch component that provides an indication of pitch angle of the head mounted display.

7. The augmented reality surgical system of claim 6 , wherein the computer equipment is configured to set an origin reference point for measurement of a yaw component of the head motion signal based on processing the yaw component of the head motion signal during times when the pitch component of the head motion signal is below a defined threshold value.

8. The augmented reality surgical system of claim 7 , wherein the motion sensor is a sourceless orientation sensor, and the computer equipment is configured to correct yaw drift of the sourceless orientation sensor towards zero responsive to determining that the pitch component of the head motion signal is below the defined threshold value.

9. The augmented reality surgical system of claim 8 , wherein the sourceless orientation sensor comprises a gyroscope, and the computer equipment is configured to estimate biases of the gyroscope giving rise to the yaw drift, using yaw drift correction pseudo-measurements determined while the pitch component of the head motion signal is below the defined threshold value.

10. The augmented reality surgical system of claim 6 , wherein the computer equipment is configured to measure drift in a yaw component of the head motion signal while a static image is displayed on the display screen, and to then compensate the head motion signal to remove the measured drift in the yaw component.

11. The augmented reality surgical system of claim 1 , wherein the computer equipment selects a primary one of the video streams that is output as a video signal to the display screen of the head mounted display based on a pitch component of the head motion signal being less than a first threshold value.

12. The augmented reality surgical system of claim 11 , wherein the computer equipment selects a secondary one of the video streams that is output as a video signal to the display screen of the head mounted display based on the pitch component of the head motion signal being above the first threshold value and below a second threshold value.

13. The augmented reality surgical system of claim 1 , wherein the computer equipment recognizes voice commands and/or gesture commands from the user, and defines and/or adjusts the yaw and pitch coordinates of the virtual floating panels displayed through the display screen responsive to the recognized voice commands and/or gesture commands.

14. The augmented reality surgical system of claim 1 , further comprising a gesture sensor connected to the head mounted display and configured to provide a gesture signal to the computer equipment, wherein the computer equipment is configured to identify a gesture made by the user responsive to the gesture signal and perform a command associated with the identified gesture, the command controls which of the video streams are output as a video signal to the display screen.

15. The augmented reality surgical system of claim 14 , wherein the gesture sensor comprises a video camera that outputs the gesture signal to the computer equipment.

16. The augmented reality surgical system of claim 14 , wherein the gesture sensor comprises a photoelectric motion sensor and/or a proximity sensor comprising at least infrared emitter that is adjacent to at least one photodiode, the at least one photodiode senses infrared light emitted by the infrared emitter which is reflected back by an object placed by the user within a field of view of the at least one photodiode, the at least one photodiode outputting the gesture signal to the computer equipment.

17. The augmented reality surgical system of claim 14 , wherein the gesture sensor comprises one or more ultrasonic echo ranging transducers that output the gesture signal to the computer equipment responsive to bouncing an ultrasonic signal off an object placed by the user within range of the one or more ultrasonic echo ranging transducers.

18. The augmented reality surgical system of claim 1 , further comprising a video camera connected to the head mounted display to be forward-facing away from the user, wherein the computer equipment communicates video from the video camera through a network to another head mounted display worn by another person.

19. The augmented reality surgical system of claim 1 , further comprising a video camera connected to the head mounted display to be forward-facing away from

Assignees

• Globus Medical Inc

Inventors

• JONES Kenneth Milton
• Popoolapade John
• CALLOWAY Thomas
• Lemoine Thierry
• Jutteau Christian
• Bruzy Christophe
• James Yannick
• LAGUARDA Joachim
• XING Dong-Mei Pei
• Gorges Sebastien
• Yarin Paul Michael

Classifications

• G06T2219/2016

  Rotation, translation, scaling · CPC title

• G06T19/20

  Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts · CPC title

• G06F3/017

  Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883) · CPC title

• G06T2210/41

  Medical · CPC title

• A61B34/10

  Computer-aided planning, simulation or modelling of surgical operations · CPC title