Augmented reality headset for a surgical robot

Patent metadata

Field	Value
Publication number	US-12186138-B2
Application number	US-202017039949-A
Country	US
Kind code	B2
Filing date	Sep 30, 2020
Priority date	Sep 30, 2020
Publication date	Jan 7, 2025
Grant date	Jan 7, 2025

Abstract

Disclosed is an augmented reality (AR) headset that provides a wearer with spatial, system, and temporal contextual information of a surgical robotic system to guide the wearer in configuring, operating, or troubleshooting the surgical robotic system prior to, during, or after surgery. The spatial context information may be rendered to display spatially-fixed 3D-generated virtual models of the robotic arms, instruments, bed, and other components of the surgical robotic system that match the actual position or orientation of the surgical robotic system in the AR headset's coordinate frame. The AR headset may communicate with the surgical robotic system to receive real-time state information of the components of the surgical robotic system. The AR headset may use the real-time state information to display context-sensitive user interface information such as tips, suggestions, visual or audio cues on maneuvering the robotic arms and table to their target positions and orientations or for troubleshooting purpose.

First claim

The invention claimed is:

1. An augmented reality (AR) device for a surgical robotic system, comprising:
a sensor configured to capture image data of a first component of the surgical robotic system;
a processor configured to:
establish a global coordinate frame for the AR device and the surgical robotic system based on the image data for virtual components of the surgical robotic system to be created in the coordinate frame of the AR device;
receive spatial information and real-time system state information of the surgical robotic system; and
create a three-dimensional (3D) virtual model of a second component of the surgical robotic system in the coordinate frame of the AR device based on the spatial information and the real-time system state information of the surgical robotic system and based on the global coordinate frame; and

a display configured to present the 3D virtual model of the second component of the surgical robotic system.

2. The AR device of claim 1 , wherein the processor configured to establish the global coordinate frame for the AR device and the surgical robotic system comprises:
transmit, by the AR device, the image data of the first component to the surgical robotic system for the surgical robotic system to establish the global coordinate frame for the coordinate frame of the AR device and a coordinate frame of the surgical robotic system.

3. The AR device of claim 2 , wherein the spatial information of the surgical robotic system is received in the coordinate frame of the AR device.

4. The AR device of claim 1 , wherein the first component of the surgical robotic system is a fiduciary marker of the surgical robotic system, and wherein the processor configured to establish the global coordinate frame for the AR device and the surgical robotic system comprises:
analyze the image data of the fiduciary marker based on a fixed location of the fiduciary marker in a coordinate frame of the surgical robotic system to establish the global coordinate frame for the coordinate frame of the AR device for the coordinate frame of the surgical robotic system.

5. The AR device of claim 4 , wherein the spatial information of the surgical robotic system comprises spatial information of the second component in the coordinate frame of the surgical robotic system, and wherein the processor configured to create the 3D virtual model of the second component of the surgical robotic system in the coordinate frame of the AR device comprises:
translate the spatial information of the second component in the coordinate frame of the surgical robotic system to the coordinate frame of the AR device based on the global coordinate frame.

6. The AR device of claim 1 , further comprising the processor configured to maintain a position and orientation of the 3D virtual model of the second component of the surgical robotic system as the coordinate frame of the AR device changes relative to a coordinate frame of the surgical robotic system.

7. The AR device of claim 1 , wherein the spatial information of the surgical robotic system comprises spatial information of the second component, and wherein the processor configured to create the 3D virtual model of the second component comprises:
create a position and orientation of the second component of the surgical robotic system in the coordinate frame of the AR device based on the spatial information of the second component, wherein the position and orientation of the second component in the coordinate frame of the AR device matches an actual position and orientation of the second component or a target position and orientation of the second component.

8. The AR device of claim 7 , wherein the real-time system state information of the surgical robotic system comprises real-time system state information of the second component, and wherein the processor configured to create the 3D virtual model of the second component further comprises:
create context-sensitive information of the second component of the surgical robotic system based on the real-time system state information of the second component.

9. The AR device of claim 1 , wherein the second component of the surgical robotic system comprises a robotic arm or an operating table of the surgical robotic system.

10. The AR device of claim 9 , wherein the 3D virtual model of the second component comprises:
a 3D virtual rendering of a position and orientation of the robotic arm or the operating table; and
visual or audible communication of the real-time system state information of the robotic arm or the operating table.

11. A method comprising:
capturing, using a sensor of an augmented reality (AR) device, image data of a first component of a surgical robotic system;
establishing a global coordinate frame for the AR device and the surgical robotic system based on the image data for virtual components of the surgical robotic system to be created in the coordinate frame of the AR device;
receiving spatial information and real-time system state information of the surgical robotic system;
creating a three-dimensional (3D) virtual model of a second component of the surgical robotic system in the coordinate frame of the AR device based on the spatial information and the real-time system state information of the surgical robotic system and based on the global coordinate frame; and
presenting, on a display of the AR device, the 3D virtual model of the second component of the surgical robotic system.

12. The method of claim 11 , wherein establishing the global coordinate frame for the AR device and the surgical robotic system comprises transmitting, by the AR device, the image data of the first component to the surgical robotic system for the surgical robotic system to establish the global coordinate frame for the coordinate frame of the AR device and a coordinate frame of the surgical robotic system.

13. The method of claim 12 , wherein the spatial information of the surgical robotic system is received in the coordinate frame of the AR device.

14. The method of claim 11 ,
wherein the first component of the surgical robotic system is a fiduciary marker of the surgical robotic system, and
wherein establishing the global coordinate frame for the AR device and the surgical robotic system comprises analyzing the image data of the fiduciary marker based on a fixed location of the fiduciary marker in a coordinate frame of the surgical robotic system to establish the global coordinate frame for the coordinate frame of the AR device for the coordinate frame of the surgical robotic system.

15. The method of claim 14 ,
wherein the spatial information of the surgical robotic system comprises spatial information of the second component in the coordinate frame of the surgical robotic system, and
wherein creating the 3D virtual model of the second component of the surgical robotic system in the coordinate frame of the AR device comprises translating the spatial information of the second component in the coordinate frame of the surgical robotic system to the coordinate frame of the AR device based on the global coordinate frame.

16. The method of claim 11 further comprising maintaining a position and orientation of the 3D virtual model of the second component of the surgical robotic system as the coordinate frame of the AR device changes relative to a coordinate frame of the surgical robotic system.

17. The method of claim 11 , wherein the second component of the surgical robotic system comprises a robotic arm or an operating table of the surgical robotic system.

18. The method o

Assignees

• Verb Surgical Inc

Inventors

• Song Tianyu
• Olson Blade
• Fuerst Bernhard A
• Fer Danyal

Classifications

• A61B2090/365

  augmented reality, i.e. correlating a live optical image with another image · CPC title

• A61B90/37

  Surgical systems with images on a monitor during operation · CPC title

• A61B2034/301

  for introducing or steering flexible instruments inserted into the body, e.g. catheters or endoscopes · CPC title

• A61B2034/2065

  Tracking using image or pattern recognition · CPC title

• A61B2034/102

  Modelling of surgical devices, implants or prosthesis · CPC title