Systems and methods for interactive user interfaces for robotic minimally invasive surgical systems

Patent metadata

Field	Value
Publication number	US-9795446-B2
Application number	US-201313775574-A
Country	US
Kind code	B2
Filing date	Feb 25, 2013
Priority date	Jun 6, 2005
Publication date	Oct 24, 2017
Grant date	Oct 24, 2017

Abstract

In one embodiment of the invention, a method for a minimally invasive surgical system is disclosed. The method includes capturing and displaying camera images of a surgical site on at least one display device at a surgeon console; switching out of a following mode and into a masters-as-mice (MaM) mode; overlaying a graphical user interface (GUI) including an interactive graphical object onto the camera images; and rendering a pointer within the camera images for user interactive control. In the following mode, the input devices of the surgeon console may couple motion into surgical instruments. In the MaM mode, the input devices interact with the GUI and interactive graphical objects. The pointer is manipulated in three dimensions by input devices having at least three degrees of freedom. Interactive graphical objects are related to physical objects in the surgical site or a function thereof and are manipulatable by the input devices.

First claim

We claim:

1. A method for a minimally invasive teleoperated surgical system, the method comprising:
displaying one or more captured camera images of a surgical site on at least one display device at a first surgeon console, the one or more captured camera images including an image of a first tool;
overlaying a first menu system, a pointer, and a first interactive graphical tool icon onto the one or more captured camera images of the surgical site, the first interactive graphical tool icon associated with the image of the first tool in the one or more captured camera images of the surgical site;
if the pointer is moved to be over a first menu button in the first menu system in response to movement of at least one input device and the first menu button is selected by the at least one input device, overlaying a first pull down menu list of a plurality of menu options each associated with a medical image data set for selection;
if the pointer is moved to be over a second menu button in the first menu system in response to movement of at least one input device and the second menu button is selected by the at least one input device, overlaying a second menu system onto the one or more captured camera images, associated with defining and adjusting virtual fixtures to guide the movement of the first tool;
if the pointer is moved to be over the first interactive graphical tool icon in response to movement of at least one input device and the first interactive graphical tool icon is selected by the at least one input device, overlaying a second pull down menu list of a plurality of menu options each associated with the first tool;
controlling the first tool using the at least one input device to perform a teleoperated surgical procedure; and
implementing virtual fixtures to guide the movement of the first tool if virtual fixtures are selected from the second menu system.

2. The method of claim 1 , wherein
the second pull down menu list of the plurality of menu options each associated with the first tool include
a first menu option to select to capture intra-operative images with the first tool and overlay them onto the one or more captured camera images of the surgical site in a flashlight image window near the first tool, and
a second menu option to select to capture intra-operative images with the first tool and overlay them onto the one or more captured camera images of the surgical site in a first inset window.

3. The method of claim 2 , wherein
the one or more captured camera images include an image of a second tool;
a second interactive graphical tool icon is overlaid onto the one or more captured camera images of the surgical site, the second interactive graphical tool icon is associated with the image of the second tool in the one or more captured camera images of the surgical site; and
if the pointer is moved to be over the second interactive graphical tool icon in response to movement of at least one input device and the second interactive graphical tool icon is selected by the at least one input device, overlaying a third pull down menu list of a plurality of menu options each associated with the second tool.

4. The method of claim 3 , wherein
the third pull down menu list of the plurality of menu options each associated with the second tool include
a first menu option to select to capture intra-operative images with the second tool and overlay them onto the one or more captured camera images of the surgical site in a flashlight image window near the second tool, and
a second menu option to select to capture intra-operative images with the second tool and overlay them onto the one or more captured camera images of the surgical site in a second inset window.

5. The method of claim 2 , wherein
if the pointer is moved to be over a fourth menu button in the first menu system in response to movement of at least one input device and the fourth menu button is selected by the at least one input device, overlaying a fourth menu system indicating menu options associated with a plurality of slave robots controllable by the first surgeon console.

6. The method of claim 5 , further comprising:
selecting a first menu option associated with a first slave robot controllable by the first surgeon console; and
adapting the first surgeon console to control the first slave robot.

7. The method of claim 6 , further comprising:
selecting a second menu option associated with a second slave robot controllable by the first surgeon console; and
adapting the first surgeon console to control the second slave robot.

8. A minimally invasive teleoperated surgical system, the system comprising:
a display device at a first surgeon console to display one or more captured camera images of a surgical site, the one or more captured camera images including an image of a first tool;
a first menu system, a pointer, and a first graphical tool icon to overlay onto the one or more captured camera images of the surgical site, the first graphical tool icon associated with the image of the first tool in the one or more captured camera images of the surgical site;
a first menu button in the first menu system responsive to a movement of the pointer controlled by at least one input device and the first menu button is selected by the at least one input device to overlay a first pull down menu list of a plurality of menu options each associated with a medical image data set for selection;
a second menu button in the first menu system responsive to movement of the pointer controlled by the at least one input device, the second menu button is selected by the at least one input device to overlay, onto the one or more captured camera images of the surgical site, a second menu system associated with defining and adjusting virtual fixtures to guide the movement of the first tool;
controlling the first tool using the at least one input device to perform a teleoperated surgical procedure; and
implementing virtual fixtures to guide the movement of the first tool if virtual fixtures are selected from the second menu system;
wherein if the pointer is moved to be over the first graphical tool icon in response to movement of the at least one input device and the first graphical tool icon is selected by the at least one input device, overlaying a second pull down menu list of a plurality of menu options each associated with the first tool.

9. The system of claim 8 , wherein
the second pull down menu list of the plurality of menu options each associated with the first tool include
a first menu option to select to capture intra-operative images with the first tool and overlay them onto the one or more captured camera images of the surgical site in a flashlight image window near the first tool, and
a second menu option to select to capture intra-operative images with the first tool and overlay them onto the one or more captured camera images of the surgical site in a first inset window.

10. The system of claim 9 , wherein
the one or more captured camera images include an image of a second tool;
a second graphical tool icon is overlaid onto the one or more captured camera images of the surgical site, the second graphical tool icon is associated with the image of the second tool in the one or more captured camera images of the surgical site; and
if the pointer is moved to be over the second graphical tool icon in response to movement of at least one input device and the second graphical tool icon is selected by the at least one input device, overlaying a third pull down menu list of a plurality of menu options each associated with the second tool.

11. The system of claim 10 , wherein
the third pull down menu list of t

Assignees

• Intuitive Surgical Operations
• Univ Johns Hopkins

Inventors

• Dimaio Simon P
• Hasser Christopher J
• Taylor Russell H
• Larkin David Q
• Kazanzides Peter
• Deguet Anton
• Vagvolgyi Balazs Peter
• Leven Joshua

Classifications

• A61B2090/365

  augmented reality, i.e. correlating a live optical image with another image · CPC title

• A61B34/37

  Leader-follower robots (A61B34/35 takes precedence) · CPC title

• A61B34/76

  Manipulators having means for providing feel, e.g. force or tactile feedback · CPC title

• A61B34/70

  Manipulators specially adapted for use in surgery · CPC title

• A61B34/71

  Manipulators operated by drive cable mechanisms · CPC title