Robotic system providing user selectable actions associated with gaze tracking

Patent metadata

Field	Value
Publication number	US-11747895-B2
Application number	US-201414210840-A
Country	US
Kind code	B2
Filing date	Mar 14, 2014
Priority date	Mar 15, 2013
Publication date	Sep 5, 2023
Grant date	Sep 5, 2023

Abstract

A medical robotic system includes a viewer, a gaze tracker, and a processor programmed to: draw an area or volume defining shape overlaid on an image based on the tracked gaze point after the user has gazed on the tracked gaze point for a programmed period of time; in response to receiving a user-selected action command, assign a fixed virtual constraint to the area or volume defining shape and constrain movement of a robotic tool; map points of the robotic tool in a tool reference frame to a viewer reference frame; determine a closest object to the tracked gaze point is the robotic tool based at least in part on the mapped points; display an object including text identifying the robotic tool, overlaid on the image, proximate to the robotic tool based on determining the robotic tool is the closest object; and perform an action indicated by the object.

First claim

What is claimed is:

1. A medical robotic system comprising:
a viewer for displaying an image of a work site;
a gaze tracker for tracking a gaze point of a user on the viewer; and
a processor programmed to:
draw an area or volume defining shape, overlaid on the image, in a position determined by the gaze tracker based on the tracked gaze point on the viewer after the user has gazed on the tracked gaze point on the viewer for a programmed period of time;
in response to receiving a user-selected action command, assign a fixed virtual constraint to the area or volume defining shape overlaid on the image and constrain movement of a robotic tool according to the fixed virtual constraint;
map points of the robotic tool in a tool reference frame to a viewer reference frame of the viewer;
determine a closest object to the tracked gaze point is the robotic tool based at least in part on the mapped points;
display an object, at least partially overlaid on the image, proximate to the robotic tool based on determining the robotic tool is the closest object, wherein the object includes text identifying the robotic tool; and
perform an action indicated by the object at least partially overlaid on the image.

2. The medical robotic system of claim 1 , wherein the processor is programmed to:
determine an association between the gaze point and the robotic tool;
identify the robotic tool;
identify an input device associated with the robotic tool;
disassociate the robotic tool from the input device and associate another robotic tool to the input device; and
automatically track movement of the robotic tool to maintain an image of the robotic tool within a field of view of an image capturing device capturing images of the work site.

3. The medical robotic system of claim 1 , wherein the text is displayed in a menu of available ones of a plurality of user selectable actions.

4. The medical robotic system of claim 1 , wherein the processor is programmed to:
highlight, in the image, the object closest to the tracked gaze point as determined by the gaze tracker on the viewer after the user has gazed on the viewer for a second programmed period of time, wherein the tracked gaze point does not overlap with the object closest to the tracked gaze point.

5. The medical robotic system of claim 4 , wherein the processor is programmed to adjust a gaze point calibration of the gaze tracker when an indication for such adjustment is received after highlighting the object closest to the tracked gaze point on the viewer.

6. The medical robotic system of claim 1 , wherein the gaze point is associated with the area or volume defining shape, and wherein the processor is further programmed to:
adjust an image characteristic of a region of interest defined by the area or volume defining shape; and
display an image of the area or volume defining shape on an auxiliary viewer which is also displaying the image of the work site.

7. The medical robotic system of claim 1 , wherein the processor is programmed to constrain movement of the robotic tool such that the robotic tool is limited to moving outside an area of the area or volume defining shape by preventing the robotic tool from entering a boundary defined by the area or volume defining shape.

8. The medical robotic system of claim 1 , wherein the processor is programmed to constrain movement of the robotic tool such that the robotic tool is limited to moving within an area of the area or volume defining shape by preventing the robotic tool from exiting a boundary defined by the area or volume defining shape.

9. The medical robotic system of claim 1 , wherein the processor is programmed to adjust one of a size or a location of the area or volume defining shape drawn around the tracked gaze point when an indication for such adjustment is received after causing the area or volume defining shape to be drawn around the tracked gaze point on the viewer.

10. The medical robotic system of claim 1 , wherein the gaze point is associated with a cursor, and the processor is programmed to:
display the cursor at the gaze point on the viewer;
store a current position of the gaze point in memory; and
perform an action related to a trajectory of the gaze point over the viewer.

11. The medical robotic system of claim 10 , wherein the processor is programmed to:
cause the cursor to be automatically displayed at the tracked gaze point as determined by the gaze tracker on the viewer after the user has gazed on the viewer for a third programmed period of time.

12. The medical robotic system of claim 11 , wherein the processor is programmed to adjust a gaze point calibration of the gaze tracker when an indication for such adjustment is received after causing the cursor to be drawn at the tracked gaze point on the viewer.

13. The medical robotic system of claim 1 , wherein the processor is programmed to identify a user selection from among a plurality of user selectable actions by processing a voice command received at a microphone.

14. The medical robotic system of claim 1 , wherein the processor is programmed to identify a user selection from among a plurality of user selectable actions by processing a command received at a user interface input.

15. The medical robotic system of claim 1 , wherein constraining movement of the robotic tool according to the fixed virtual constraint comprises limiting a working end of the robotic tool to moving within the area or volume defining shape while allowing a portion of the robotic tool disposed between the working end and a robotic arm of the medical robotic system to move outside the area or volume defining shape.

16. A medical robotic system comprising:
a viewer for displaying an image of a work site;
a gaze tracker for tracking a gaze point of a user on the viewer; and
a processor programmed to:
draw an area or volume defining shape, overlaid on the image, in a position determined by the gaze tracker based on the tracked gaze point on the viewer after the user has gazed on the tracked gaze point on the viewer for a programmed period of time;
in response to receiving a user-selected action command, assign a fixed virtual constraint to the area or volume defining shape overlaid on the image and constrain movement of a robotic tool according to the fixed virtual constraint;
highlight an object closest to the tracked gaze point on the viewer; and
initiate a calibration function after highlighting the object to adjust a gaze point calibration of the gaze tracker when an indication for such adjustment is received after causing the area or volume defining shape to be drawn around the tracked gaze point on the viewer.

17. The medical robotic system of claim 16 , wherein the gaze point is associated with a calibration function, and the processor is programmed to calibrate the tracked gaze point as determined by the gaze tracker with an actual gaze point of the user on the viewer.

18. The medical robotic system of claim 17 , wherein the processor is programmed to calibrate the tracked gaze point as determined by the gaze tracker with the actual gaze point of the user on the viewer by sequentially displaying markers of a known position on the viewer and adjusting for errors between the known position and the tracked gaze point as determined by the gaze tracker.

19. A medical robotic system comprising:
a viewer for displaying an image of a work site;
a gaze tracker for tracking a gaze point of a user on the viewer; and
a processor programmed to:
draw a box, overlaid on the image, i

Assignees

• Intuitive Surgical Operations

Inventors

• Zhao Wenyi
• Hasser Christopher J
• Itkowitz Brandon D
• Lilagan Paul E
• Scott David D
• Dimaio Simon P
• Robinson David W
• Zhao Tao

Classifications

• G06F3/013 Primary

  Eye tracking input arrangements (G06F3/015 takes precedence) · CPC title

• G06F3/04815

  Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object · CPC title