Augmented reality headset with varied opacity for navigated robotic surgery

Patent metadata

Field	Value
Publication number	US-12336868-B2
Application number	US-202418643344-A
Country	US
Kind code	B2
Filing date	Apr 23, 2024
Priority date	Dec 10, 2019
Publication date	Jun 24, 2025
Grant date	Jun 24, 2025

Abstract

A surgical system includes an AR headset and a AR headset controller. The AR headset is configured to be worn by a user during a surgical procedure and has a see-through display screen configured to display an AR image and to allow at least a portion of a real-world scene to pass therethrough for viewing by the user. The AR headset also includes an opacity filter positioned between at least one of the user's eyes and the real-world scene when the see-through display screen is viewed by the user. The opacity filter provides opaqueness to light from the real-world scene. The AR headset controller communicates with a navigation controller to receive navigation information from the navigation controller which provides guidance to the user during the surgical procedure on an anatomical structure, and generates the AR image based on the navigation information for display on the see-through display screen.

First claim

The invention claimed is:

1. A surgical system comprising:
an augmented reality (AR) headset configured to be worn by a user during a surgical procedure and including:
an AR transmitter configured to project light for an AR image;
a see-through display screen configured to combine the projected AR image and a real-world scene for viewing by the user, and
an opacity filter positioned between at least one of the user's eyes and the real-world scene when the see-through display screen is viewed by the user, wherein the

opacity filter is configured to provide opaqueness to light from the real-world scene; and
an AR headset controller configured to communicate with a navigation controller to receive navigation information from the navigation controller which provides guidance to the user during the surgical procedure on an anatomical structure, and configured to generate the AR image based on the navigation information for display on the see-through display screen,
wherein the AR headset controller is configured to electrically control an opacity to light transmissivity of at least one area of the opacity filter,
wherein the opacity filter is configured with at least two horizontally extending bands. each band extending across the entire width of the see-through display screen, and
wherein the augmented reality headset is operationally coupled to a robotic system having a robotic arm that is controllable based on hand gesture commands that are sensed by the augmented reality headset.

2. The surgical system of claim 1 , wherein the see-through display screen includes a curved lens that extends laterally across and downward from a front part of the AR headset, and the opacity filter is on a surface of the curved lens.

3. The surgical system of claim 1 , wherein:
the AR headset controller is configured to display in a region of the see-through display screen aligned with the laterally extending band of the opacity filter at least one of: 2D axial, sagittal, and/or coronal view images of patient anatomy; a planned and/or currently tracked surgical tool pose; graphical model of surgical implant location; video from a medical instrument; and user selectable menu items triggering operations controlling medical equipment, and to display in another region of the see-through display screen that is not aligned with the laterally extending band of the opacity filter at least one of: a 3D graphical model of the anatomical structure and surgical planning information; 3D graphical model of a surgical instrument; animated 3D graphical model of a surgical instrument displayed with a pose relative to a graphical model of the anatomical structure that is modified to track in real-time measured poses of the surgical instrument relative to the anatomical structure; and a graphical model of the anatomical structure and the navigation information from the navigation controller which provides visual guidance to the user during the surgical procedure on the anatomical structure.

4. The surgical system of claim 1 , wherein:
wherein the see-through display screen includes a combiner configured to combine the light of the AR images projected from the AR transmitter and light from the real-world scene into a combined overlay image viewable by the user.

5. The surgical system of claim 1 , further comprising:
the AR transmitter configured to project light of the AR images responsive to signaling from the AR headset controller,
wherein the see-through display screen includes a combiner configured to combine light of the AR images projected from the AR transmitter and light from the real-world scene into a combined image viewable by the user, and
wherein the opacity filter includes an opacity gradient that extends on a surface of the combiner.

6. The surgical system of claim 1 , wherein for each of a plurality of types of AR images the AR headset controller is configured to select among a plurality of different regions of the see-through display screen for displaying the type of AR image based on a characteristic of the type of AR image and levels of opaqueness of regions of the opacity filter aligned with the different regions of the see-through display screen.

7. The surgical system of claim 1 , wherein the AR headset further comprises a camera configured to output video frames capturing the real-world scene, and the AR headset controller is configured to determine brightness and/or color of a defined area within the video frames and electrically control an opacity to light transmissivity of at least one area of the opacity filter based on the determined brightness and/or color of the defined area within the video frames.

8. The surgical system of claim 1 , wherein the AR headset further comprises a camera configured to output video frames capturing the real-world scene, and the AR headset controller is configured to determine brightness and/or color of a defined area within the video frames and control brightness and/or color of the AR image displayed on the see-through display screen based on the determined brightness and/or color of the defined area within the video frames.

9. The surgical system of claim 1 , wherein the AR headset controller is configured to determine poses of the AR headset and electrically control an opacity to light transmissivity of at least one area of the opacity filter based on the determined poses of the AR headset.

10. The surgical system of claim 9 , wherein the AR headset controller is configured to determine the poses of the AR headset relative to a surgical site containing the anatomical structure and electrically control an opacity to light transmissivity of at least one area of the opacity filter based on the determined poses of the AR headset relative to the surgical site.

11. The surgical system of claim 1 , wherein the AR headset further comprises a camera configured to output video frames capturing the real-world scene, wherein the AR headset controller is configured to identify an object within the video frames, determine brightness and/or color of the identified object within the video frames, determine poses of the AR headset relative to a real-world object corresponding to the identified object within the video frames, and electrically control an opacity to light transmissivity of at least one area of the opacity filter based on the determined brightness and/or color of the identified object within the video frames and based on the determined poses.

12. The surgical system of claim 1 , wherein the AR headset further comprises a camera configured to output video frames capturing the real-world scene, wherein the AR headset controller is configured to identify an object within the video frames, determine brightness and/or color of the identified object within the video frames, determine poses of the AR headset relative to a real-world object corresponding to the identified object within the video frames, and to control brightness and/or color of the AR image displayed on the see-through display screen based on the determined brightness and/or color of the identified object within the video frames and based on the determined poses.

13. The surgical system of claim 1 , wherein:
the opacity filter includes a plurality of pixels arranged in a grid and electrically controllable to adjust pixel opacity to light transmissivity; and
wherein the AR headset controller is configured to increase an opacity of at least some pixels that are selected among the grid based on being aligned with or offset a distance from at least a portion of a shape of an AR image displayed on the see-through display screen.

14. The surgical system of claim

Assignees

• Globus Medical Inc

Inventors

• CALLOWAY Thomas
• Healy Weston
• Dulin Isaac
• Earle Dale
• Matsuda Keiichi

Classifications

• G06V20/20

  in augmented reality scenes · CPC title

• G06T2215/16

  Using real world measurements to influence rendering · CPC title

• G02B27/0172

  characterised by optical features · CPC title

• G06T7/74

  involving reference images or patches · CPC title

• A61B2090/365

  augmented reality, i.e. correlating a live optical image with another image · CPC title