User input device for use in robotic surgery

Patent metadata

Field	Value
Publication number	US-11534246-B2
Application number	US-201916439591-A
Country	US
Kind code	B2
Filing date	Jun 12, 2019
Priority date	Jun 15, 2018
Publication date	Dec 27, 2022
Grant date	Dec 27, 2022

Abstract

User input devices (UIDs) for controlling a surgical robotic system are described. A UID can include one or more tracking sensors to generate respective spatial state signals in accordance with a pose of the UID. At least one of the tracking sensors can be a camera. In the case of multiple tracking sensors, the spatial state signals are processed by a sensor fusion algorithm to generate a more robust, single tracking signal and a quality measure. The tracking signal and the quality measure are then used by a digital control system to control motion of a surgical robotic system actuator that is associated with the UID. Other embodiments are also described and claimed.

First claim

What is claimed is:

1. A user input device (UID) for a surgical robotic system, comprising:
a device body extending from a proximal end to a distal end, the device body configured to be held by an operator's hand for teleoperation of a surgical robotic arm with the distal end of the device body being forward of the proximal end, the proximal end is to be cupped in the operator's hand, and a plurality of grip linkages mounted on the device body around a central axis of the device body and configured to be manipulated by the operator's hand for teleoperation of an end effector coupled to the surgical robotic arm;
a camera having an image sensor and an imaging lens, the camera being mounted on the device body such that the imaging lens faces forward at the distal end of the device body to view a marker; and
a UID controller coupled to the image sensor, wherein the UID controller is to send digital images of the marker produced by the image sensor to a processor that computes a visual modality estimation of a pose of the UID based on detecting the marker in the digital images.

2. The UID of claim 1 further comprising a first tracking sensor mounted on the device body, wherein the first tracking sensor is configured to generate a first spatial state signal in response to movement of the device body and the UID controller is coupled to the first tracking sensor and configured to send the first spatial state signal from the first tracking sensor to the processor, and wherein the processor is to combine information from the visual modality estimation and the first spatial state signal to produce a tracking signal indicative of the pose of the UID.

3. The UID of claim 2 wherein the first tracking sensor is an electromagnetic, EM, tracking sensor.

4. The UID of claim 2 wherein the first tracking sensor is an inertial measurement unit, IMU, and the first spatial state signal comprises accelerometer measurements and gyroscope measurements.

5. The UID of claim 4 further comprising a second tracking sensor mounted on the device body and being an electromagnetic (EM) tracking sensor that is to generate a second spatial state signal in accordance with the pose of the UID and in six degrees of freedom, wherein the processor is to produce the tracking signal by combining information from the visual modality estimation, the first spatial state signal and the second spatial state signal.

6. The UID of claim 1 wherein the plurality of grip linkages are positioned behind the camera.

7. The UID of claim 2 further comprising
a serial data communications cable connected to the UID controller to transmit i) the digital images of the marker produced by the image sensor and ii) the first spatial state signal from the first tracking sensor, to the processor.

8. The UID of claim 7 further comprising a second tracking sensor mounted on the device body and being an electromagnetic (EM) tracking sensor that is to generate a second spatial state signal in accordance with the pose of the UID and in six degrees of freedom, wherein the processor is to produce the tracking signal by combining information from the visual modality estimation, the first spatial state signal and the second spatial state signal.

9. A user input device (UID) for a surgical robotic system, comprising:
a device body extending from a proximal end to a distal end, the device body configured to be held by an operator's hand for teleoperation of a surgical robotic arm with the distal end being forward of the proximal end and the proximal end is to be cupped in the operator's hand;
a camera having an image sensor and an imaging lens, the camera being mounted on the device body such that the imaging lens faces forward at the distal end of the device body to view a marker comprising a plurality of infrared light sources positioned on a user console of the surgical robotic system;
an electromagnetic, EM, tracking sensor mounted on the device body and that is to generate a first spatial state signal in accordance with the pose of the UID; and
a UID controller coupled to the image sensor and the EM tracking sensor, wherein the UID controller is to send digital images of the marker produced by the image sensor and the first spatial state signal generated by the EM tracking sensor to a processor, wherein the processor is to compute a visual modality estimation of a pose of the UID, based on detecting the marker in the digital images, and combine the visual modality estimation with the first spatial signal to produce a tracking signal indicative of the pose of the UID.

10. The UID of claim 9 further comprising an inertial measurement unit, IMU, mounted on the device body, wherein the IMU is to produce a second spatial state signal in accordance with the pose of the UID, and the UID controller is coupled to the IMU and configured to send the first spatial state signal, the second spatial state signal, and the digital images of the marker to the processor, and wherein the processor is to produce the tracking signal by combining information from the visual modality estimation, the first spatial state signal, and the second spatial state signal.

11. The UID of claim 10 in combination with the processor computing a quality measure that is indicative of a discrepancy between i) EM tracking sensor measurements in the first spatial state signal and ii) IMU measurements in the second spatial state signal.

12. A method for teleoperation control of a surgical robotic arm in a surgical robotic system, the method comprising:
generating by a camera that is looking outward from a user input device, UID, of a surgical robotic system, a plurality of digital images that capture a marker comprising a plurality of infrared light sources positioned on a user console of the surgical robotic system;
digitally processing the digital images to detect an object in the digital images that corresponds to the marker;
interpreting the detected object to compute a visual modality estimation of a pose of the UID, wherein the visual modality estimation tracks movement of the UID; and
generating actuator control commands to control the surgical robotic arm based on the visual modality estimation, to cause movement of an end effector coupled to the surgical robotic arm that mimics the movement of the UID.

13. The method of claim 12 further comprising
generating by an electromagnetic, EM, tracking sensor a first spatial state signal in accordance with the pose of the UID; and
combining information from the visual modality estimation and from the first spatial state signal to produce a tracking signal indicative of the pose of the UID, wherein generating the actuator control commands is based on the tracking signal.

14. The method of claim 12 further comprising:
generating by an inertial measurement unit, IMU, in the UID a first spatial state signal in accordance with the pose of the UID; and
fusing, by the processor, information from the visual modality estimation and from the first spatial state signal, to produce a tracking signal indicative of the pose of the UID, wherein generating the actuator control commands is based on the tracking signal.

15. The method of claim 14 wherein the first spatial state signal comprises accelerometer measurements and gyroscope measurements.

16. The method of claim 14 further comprising:
generating by an electromagnetic (EM) tracking sensor in the UID a second spatial state signal in accordance with the pose of the UID and in six degrees of freedom;
fusing by the processor information from i) the visual modality estimation, ii) the first spatial stat

Assignees

• Verb Surgical Inc

Inventors

• Fuerst Bernhard Adolf
• Garcia Kilroy Pablo E
• Gonenc Berk
• Cordoba Jose Luis
• Savall Joan
• Barthel Alexander

Classifications

• A61B34/25 Primary

  User interfaces for surgical systems · CPC title

• A47C7/723

  with display screens · CPC title

• G06F3/0346

  with detection of the device orientation or free movement in a three-dimensional [3D] space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors · CPC title

• A61B2034/2057

  Details of tracking cameras · CPC title

• B25J9/1694

  characterised by use of sensors other than normal servo-feedback from position, speed or acceleration sensors, perception control, multi-sensor controlled systems, sensor fusion · CPC title