System and method for assisting operator engagement with input devices

Patent metadata

Field	Value
Publication number	US-11500472-B2
Application number	US-202117147386-A
Country	US
Kind code	B2
Filing date	Jan 12, 2021
Priority date	Jan 18, 2018
Publication date	Nov 15, 2022
Grant date	Nov 15, 2022

Abstract

A system and method of assisting operator engagement with input devices includes a repositionable structure configured to support a physical input device, a hand detection system, and a control unit. The repositionable structure includes one or more actuators. The physical input device is configured to be operated by a hand of an operator. The control unit is configured to detect the hand of the operator using the hand detection system and in response to a trigger condition, command the one or more actuators to move the input device from a first orientation to a second orientation, wherein the second orientation is closer to a grasping orientation of the hand than the first orientation is to the grasping orientation of the hand.

First claim

What is claimed is:

1. A system comprising:
a repositionable structure configured to support a physical input device, the repositionable structure comprising one or more actuators, and the physical input device configured to be operated by a hand of an operator;
a hand detection system; and
a control unit configured to:
determine a rotational orientation of the hand of the operator using the hand detection system; and
in response to a trigger condition, command the one or more actuators to physically move the input device from a first rotational orientation to a second rotational orientation, wherein the second rotational orientation is determined based on the rotational orientation of the hand, and wherein moving the input device from the first rotational orientation to the second rotational orientation moves the input device toward the rotational orientation of the hand.

2. The system of claim 1 , wherein the control unit is further configured to:
in response to the trigger condition, command the input device to move from a first position to a second position, wherein the second position is closer to a grasping position of the hand than the first position is to the grasping position of the hand.

3. The system of claim 1 , wherein the control unit is further configured to:
determine the rotational orientation of the hand based on one or more images of the hand.

4. The system of claim 1 , wherein:
the input device is associated with an end effector; and
the control unit is further configured to:
in response to detecting an engagement of the hand with the input device, command a motion of the end effector based on an input received from the input device.

5. The system of claim 1 , wherein the control unit is further configured to:
in response to detecting an engagement of the hand with the input device, command the input device to move from the second rotational orientation to a third rotational orientation, wherein the third rotational orientation is based on a rotational orientation of an end effector associated with the input device.

6. The system of claim 1 , wherein to detect the trigger condition the control unit is configured to:
detect a gesture of the hand; or
detect that the hand is less than a threshold distance from the input device; or
detect a gaze command.

7. The system of claim 1 , further comprising:
a display system, wherein the control unit is further configured to provide one or more position or orientation cues on the display system;
wherein the one or more position or orientation cues include:
images of the hand and the input device; or
an image of an end effector indication and a virtual hand, the end effector indication comprising a view of an end effector or a view of a virtual end effector, the virtual hand being positioned and oriented in the image relative to the end effector indication based on a relative position and orientation between the hand and the input device; or
an image blending an image of the hand with an image of the end effector.

8. The system of claim 1 , wherein the control unit is further configured to detect whether the hand is engaged with the input device by detecting whether a difference between an actual position of the input device and a position of the input device commanded by the control unit is greater than a threshold position difference.

9. The system of claim 1 , wherein:
the control unit is further configured to, in response to detecting an abort condition, abort the command to move the input device from the first rotational orientation to the second rotational orientation; and
to detect the abort condition the control unit is configured to:
detect an abort gesture of the hand; or
detect a lack of a recognizable gesture of the hand within a time-out period; or
determine that a threshold period of time has elapsed without detecting that the hand is engaged with the input device; or
detect that a difference between the rotational orientation of the hand and a rotational orientation of the input device is greater than a threshold rotational orientation difference; or
detect that the hand is farther away than a threshold distance from the input device.

10. The system of claim 1 , wherein the control unit is further configured to reduce a stiffness of the input device while the input device is being commanded to move from the first rotational orientation to the second rotational orientation.

11. A method of operating a system comprising a repositionable structure, a hand detection system, and a control unit, wherein the repositionable structure comprises one or more actuators and is configured to support a physical input device, the method comprising:
detecting, using the hand detection system, a rotational orientation of a hand of an operator; and
in response to a trigger condition, commanding, by the control unit, the one or more actuators to move the physical input device from a first rotational orientation to a second rotational orientation, wherein the second rotational orientation is determined based on the rotational orientation of the hand, and wherein moving the input device from the first rotational orientation to the second rotational orientation moves the input device toward the rotational orientation of the hand.

12. The method of claim 11 , further comprising:
in response to the trigger condition, commanding the input device to move from a first position to a second position, wherein the second position is closer to a grasping position of the hand than the first position is to the grasping position of the hand.

13. The method of claim 11 , further comprising: determining the rotational orientation of the hand based on a one or more images of the hand.

14. The method of claim 11 , wherein the input device is associated with an end effector, the method further comprising:
in response to detecting an engagement of the hand with the input device, commanding a motion of the end effector based on an input received from the input device.

15. The method of claim 11 , further comprising:
in response to detecting an engagement of the hand with the input device, commanding the input device to move from the second rotational orientation to a third rotational orientation, wherein the third rotational orientation is based on a rotational orientation of an end effector associated with the input device.

16. The method of claim 11 , wherein detecting the trigger condition comprises:
detecting a gesture of the hand; or
detecting that the hand is less than a threshold distance from the input device; or
detecting a gaze command.

17. The method of claim 11 , further comprising:
in response to detecting an abort condition, aborting the commanding to move the input device from the first rotational orientation to the second rotational orientation;
wherein detecting the abort condition comprises:
detecting an abort gesture of the hand; or
detecting a lack of a recognizable gesture of the hand within a time-out period; or
detecting that the hand is farther away than a threshold distance from the input device; or
detecting that a difference between the rotational orientation of the hand and a rotational orientation of the input device is greater than a threshold rotational orientation difference; or
determining that a threshold period of time has elapsed without detecting that the hand is engaged with the input device.

18. The method of claim 11 , further comprising reducing a stiffness of the input device while the input device is being commande

Assignees

• Intuitive Surgical Operations

Inventors

• He Xingchi
• Itkowitz Brandon D

Classifications

• G05B2219/39451

  Augmented reality for robot programming · CPC title

• G06F3/0346

  with detection of the device orientation or free movement in a three-dimensional [3D] space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors · CPC title

• G06F3/014

  Hand-worn input/output arrangements, e.g. data gloves · CPC title

• B25J9/1664

  characterised by motion, path, trajectory planning · CPC title

• G06F3/011 Primary

  Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00) · CPC title