Method and apparatus for hand gesture control in a minimally invasive surgical system

Patent metadata

Field	Value
Publication number	US-9901402-B2
Application number	US-201514629685-A
Country	US
Kind code	B2
Filing date	Feb 24, 2015
Priority date	Sep 21, 2010
Publication date	Feb 27, 2018
Grant date	Feb 27, 2018

Abstract

In a minimally invasive surgical system, a hand tracking system tracks a location of a sensor element mounted on part of a human hand. A system control parameter is generated based on the location of the part of the human hand. Operation of the minimally invasive surgical system is controlled using the system control parameter. Thus, the minimally invasive surgical system includes a hand tracking system. The hand tracking system tracks a location of part of a human hand. A controller coupled to the hand tracking system converts the location to a system control parameter, and injects into the minimally invasive surgical system a command based on the system control parameter.

First claim

We claim:

1. A method of operating a controller of a teleoperated system comprising:
receiving, by the controller from a hand tracking unit of the teleoperated system, a plurality of locations on a human hand;
selecting a hand gesture pose from a plurality of hand gesture poses, the selecting of the hand gesture pose being by the controller of the teleoperated system based on the plurality of locations on the human hand, a first hand gesture pose of the plurality of hand gesture poses being used to generate a plurality of hand gesture trajectories, each of the other hand gesture poses of the plurality of hand gesture poses being assigned to a corresponding one of a first plurality of system modes of operation of the teleoperated system;
determining whether the selected hand gesture pose is the first hand gesture pose used in the plurality of hand gesture trajectories, the determining being by the controller;
selecting a hand gesture trajectory from the plurality of hand gesture trajectories on a condition that the determining finds that the selected hand gesture pose is the first hand gesture pose, the selecting of the hand gesture trajectory being by the controller based on the plurality of locations, each hand gesture trajectory of the plurality of hand gesture trajectories being assigned to a corresponding one of a second plurality of system modes of operation of the teleoperated system;
changing a mode of operation of the teleoperated system to a mode of the first plurality of system modes of operation corresponding to the selected hand gesture pose on a condition that the selected hand gesture pose is not the first hand gesture pose, the changing a mode of operation of the teleoperated system to a mode of the first plurality of system modes of operation being by the controller and based on the selected hand gesture pose; and
changing the mode of operation of the teleoperated system to a mode of the second plurality of system modes of operation corresponding to the selected hand gesture trajectory on a condition that the selected hand gesture pose is the first hand gesture pose, the changing the mode of operation of the teleoperated system to a mode of the second plurality of system modes of operation being by the controller based on the selected hand gesture trajectory.

2. The method of claim 1 , wherein the changing a mode of operation of the teleoperated system to a mode of the first plurality of system modes of operation comprises:
initiating control of a user interface of the teleoperated system.

3. The method of claim 1 , wherein the changing a mode of operation of the teleoperated system to a mode of the first plurality of system modes of operation comprises:
activating a proxy visual system loop.

4. The method of claim 1 , wherein the selecting the hand gesture pose comprises:
generating an observed feature set from the plurality of locations.

5. The method of claim 4 , wherein the selecting the hand gesture pose further comprises:
comparing the observed feature set with feature sets of the plurality of hand gesture poses.

6. The method of claim 5 , wherein the selecting the hand gesture pose further comprises:
selecting the hand gesture pose based on a comparison of the observed feature set with feature sets of the plurality of hand gesture poses.

7. The method of claim 1 , wherein the selecting the hand gesture trajectory comprises:
generating, by the controller, a velocity sequence from a trajectory of the plurality of locations, wherein the velocity sequence comprises a sequence of unit velocity vectors; and
converting, by the controller, the velocity sequence into a symbol sequence, wherein the symbol sequence comprises a sequence of discrete symbols.

8. The method of claim 7 , wherein the selecting the hand gesture trajectory further comprises:
analyzing, by the controller, the symbol sequence with a plurality of Hidden Markov Models, wherein each of the plurality of hand gesture trajectories has a Hidden Markov Model in the plurality of Hidden Markov Models.

9. The method of claim 8 , wherein the selecting the hand gesture trajectory further comprises:
selecting a hand gesture trajectory based on an analysis of the symbol sequence with a plurality of Hidden Markov Models.

10. The method of claim 9 , wherein the changing the mode of operation of the teleoperated system to a mode of the second plurality of system modes of operation comprises:
mapping the selected hand gesture trajectory to a system event.

11. The method of claim 1 , wherein the changing the mode of operation of the teleoperated system to a mode of the first plurality of system modes of operation comprises:
mapping the hand gesture pose to a system event.

12. The method of claim 11 , wherein the changing the mode of operation of the teleoperated system to a mode of the first plurality of system modes of operation further comprises:
injecting the system event in the teleoperated system.

13. The method of claim 1 , wherein the changing the mode of operation of the teleoperated system to a mode of the second plurality of system modes of operation comprises:
mapping the selected hand gesture trajectory to a system event.

14. A method of operating a controller of a teleoperated system comprising:
changing a mode of operation of the teleoperated system to one of a first plurality of system modes of operation corresponding to a detected hand gesture pose on a condition that the detected hand gesture pose is not a hand gesture pose used in a plurality of hand gesture trajectories, the changing to the one of the first plurality of system modes being by the controller of the teleoperated system based on the detected hand gesture pose, the detected hand gesture pose being selected from a plurality of hand gesture poses based on a plurality of locations on a human hand, the plurality of locations being received, by the controller, from a hand tracking unit of the teleoperated system; and
changing the mode of operation of the teleoperated system to one of a second plurality of system modes of operation on a condition that the detected hand gesture pose is the hand gesture pose used in the plurality of hand gesture trajectories, the changing to the one of the second plurality of system modes being by the controller based on a detected hand gesture trajectory in the plurality of hand gesture trajectories, the detected hand gesture trajectory being based on the plurality of locations.

15. In a teleoperated system for which a plurality of hand gesture poses and a plurality of hand gesture trajectories are defined, a first one of the plurality of hand gesture poses being used in the plurality of hand gesture trajectories, each of the others of the plurality of hand gesture poses corresponding to a different mode of operation of a first plurality of modes of operation of the teleoperated system, and each of the plurality of hand gesture trajectories corresponding to a different mode of operation of a second plurality of modes of operation of the teleoperated system, a method executed by a controller of the teleoperated system, comprising:
on a condition that a detected hand gesture pose is one of the others of the plurality of hand gesture poses, changing a mode of operation of the teleoperated system to a mode of operation of the first plurality of system modes of operation corresponding to the one of the others of the plurality of hand gesture poses;
on a condition that a detected hand gesture pose is the first one of the plurality of hand gesture poses, and on the condition that a detected hand gesture trajectory is one of

Assignees

• Intuitive Surgical Operations

Inventors

• Itkowitz Brandon D
• Dimaio Simon P
• Zhao Tao

Classifications

• B25J13/084

  Tactile sensors (in general G01L5/16, G01L5/22) · CPC title

• G05B15/02

  electric · CPC title

• A61B17/00234

  for minimally invasive surgery (A61B17/0218, A61B17/0469, A61B17/12013, A61B17/1285, A61B17/29, A61B17/320016 take precedence) · CPC title

• G05B2219/36418

  Modify trajectory by operator gesture, gesture force sensed by end effector · CPC title

• G06F3/014

  Hand-worn input/output arrangements, e.g. data gloves · CPC title