Multi-panel graphical user interface for a robotic surgical system

Patent metadata

Field	Value
Publication number	US-11071595-B2
Application number	US-201715842485-A
Country	US
Kind code	B2
Filing date	Dec 14, 2017
Priority date	Dec 14, 2017
Publication date	Jul 27, 2021
Grant date	Jul 27, 2021

Abstract

A method for a robotic surgical system includes displaying a graphical user interface on a display to a user, wherein the graphical user interface includes a plurality of reconfigurable display panels, receiving a user input at one or more user input devices, wherein the user input indicates a selection of at least one software application relating to the robotic surgical system, and rendering content from the at least one selected software application among the plurality of reconfigurable display panels.

First claim

The invention claimed is:

1. A method for a robotic surgical system that includes a handheld user input device (UID) and a robotic arm with an end effector, the method comprising:
using the handheld UID to control movement of the robotic arm with the end effector while the handheld UID is in a user's hand and while the robotic surgical system is in tele operation;
displaying a graphical user interface (“GUI”) on a display to a user, wherein the GUI includes a first display panel that contains visual content from a first software application and a second display panel that contains visual content from a second software application, wherein the visual content from the first software application is endoscopic video of a surgical site and the first display panel is larger than the second display panel; and
in response to a user input at the handheld UID, pausing the robotic arm movement;
using an eye-tracking sensor to determine that the user is focusing on the second display panel in the GUI; and
in response to 1) pausing the robotic arm movement and 2) determining that the user is focusing on the second display panel, reconfiguring the display panels to render visual content from the second software application in the first display panel and to render visual content from the first software application in the second display panel.

2. The method of claim 1 , wherein reconfiguring the display panels further comprises reconfiguring a layout of the display panels.

3. The method of claim 2 , wherein reconfiguring comprises repositioning at least one of the first and second display panels.

4. The method of claim 1 , wherein reconfiguring comprises resizing at least one of the first and second display panels.

5. The method of claim 1 , wherein the surgical site is a location at which a surgical task is to be performed by the end effector of the robotic arm.

6. The method of claim 5 , wherein the user input is a first user input, wherein the method further comprises
in response to a second user input, controlling the end effector to perform the surgical task at the surgical site that includes at least one movement of the end effector;
based on the endoscopic video, determining that the surgical task is completed by the end effector; and
in response to the completion of the surgical task
switching the handheld UID from controlling the movement of the robotic arm with the end effector to selectively controlling the GUI, and
reconfiguring the display panels to render visual content from the first software application in the first display panel and to render visual content from the second software application in the second display panel.

7. The method of claim 1 , wherein the handheld UID includes a clutch mechanism for pausing the robotic arm movement, wherein the method further comprises receiving the user input that engages the clutch mechanism to pause the robotic arm movement.

8. The method of claim 1 , wherein the handheld UID is configured to remotely control movement of the robotic arm.

9. The method of claim 7 , wherein the robotic arm movement remains paused while the clutch mechanism is engaged.

10. The method of claim 1 , further comprising mirroring at least some of the rendered content onto a second display.

11. A robotic surgical system, comprising:
a robotic surgical instrument;
a handheld user input devices (UID) configured to control movement of the robotic surgical instrument while the handheld UID is in a user's hand and while the robotic surgical system is in tele-operation;
a display configured to display a graphical user interface (GUI) including a first display panel that contains visual content from a first software application and a second display panel that contains visual content from a second software application, wherein the visual content from the first software application is endoscopic video of a surgical site and the first display panel is larger than the second display panel; and
an eye-tracking sensor configured to detect whether the user is focusing on a portion of the display,
wherein, in response to a user input at the handheld UID, the robotic surgical instrument is configured to pause in place,
wherein, in response to 1) the robotic surgical instrument pausing in place and 2) the eye-tracking sensor detecting that the user is focusing on the second display panel in the GUI,
the display becomes reconfigured to render visual content from the second software application in the first display panel and to render visual content from the first software application in the second display panel.

12. The system of claim 11 , wherein the first and second display panels are reconfigurable in layout.

13. The system of claim 11 , wherein at least one of the display panels may be reconfigured by at least one of repositioning or resizing.

14. The system of claim 11 , wherein the surgical site is a location at which a surgical task is to be performed by the robotic surgical instrument.

15. The system of claim 14 ,
wherein the user input is a first user input,
wherein the handheld UID is further configured to receive a second user input to control the robotic surgical instrument to perform the surgical task at the surgical site,
wherein the system is configured to perform a machine vision algorithm upon the endoscopic video to detect whether the surgical task is complete,
wherein, in response to the machine vision algorithm detecting that the surgical task is completed by the robotic surgical instrument,
the handheld UID is reconfigured from controlling the movement of the robotic surgical instrument to controlling the GUI, and
the display is reconfigured to render visual content from the first software application in the first display panel and to render visual content from the second software application in the second display panel.

16. The system of claim 11 , wherein, while the handheld UID is configured to selectively control the GUI in response to the user input.

17. The system of claim 11 , wherein the handheld UID includes a clutch mechanism for pausing the robotic surgical instrument in place, wherein the user input engages the clutch mechanism to cause the robotic surgical instrument to pause in place.

18. The system of claim 11 , wherein the second software application is one of an image viewer application, a procedural template application, a teleconference application, and a timer application.

19. The system of claim 11 , wherein at least some of the rendered content is mirrored on a second display.

20. The system of claim 11 , wherein the handheld UID is configured to remotely control the robotic surgical system.

21. The method of claim 1 , wherein the handheld UID is further configured to selectively control the GUI.

Assignees

• Verb Surgical Inc

Inventors

• Johnson Eric Mark
• Essock-Burns Emma
• Miller Lawrence Edward
• Brahic Francois W

Classifications

• G16H40/63

  for local operation · CPC title

• G16H20/40

  relating to mechanical, radiation or invasive therapies, e.g. surgery, laser therapy, dialysis or acupuncture · CPC title

• A61B18/1482

  having a long rigid shaft for accessing the inner body transcutaneously in minimal invasive surgery, e.g. laparoscopy · CPC title

• A61B2017/00225

  Systems for controlling multiple different instruments, e.g. microsurgical systems · CPC title

• A61B90/361

  Image-producing devices, e.g. surgical cameras · CPC title