Virtual tools for supported tele-operations

Patent metadata

Field	Value
Publication number	US-12455669-B2
Application number	US-202418593955-A
Country	US
Kind code	B2
Filing date	Mar 3, 2024
Priority date	Mar 3, 2024
Publication date	Oct 28, 2025
Grant date	Oct 28, 2025

Abstract

A user interface for instructing a tele-operating system and the tele-operating system are provided. The user interface includes an output device configured to output at least one virtual tool representation of at least one tool to an operator of the tele-operating system, and an input device configured to obtain a selection instruction from the operator for selecting one of the output at least one virtual tool representation. The user interface further includes a control circuit configured to determine a task based on the selected virtual tool representation, control displaying the selected virtual tool representation to the operator via the output device, acquire an action instruction from the operator via the input device, interpret the acquired action instruction based on the determined task, and control the tele-operating system to perform an action using the at least one tool based on the interpreted action instruction and the selected virtual tool representation.

First claim

What is claimed is:

1 . A user interface for instructing a tele-operating system to perform an action, the user interface comprising:
an output device configured to output plural virtual tool representations of a plurality of tools to an operator of the tele-operating system; an input device configured to receive a selection instruction from the operator for selecting one of the output plural virtual tool representations; a control circuit configured to determine a task on the selected virtual tool representation, control displaying the selected virtual tool representation to the operator via the output device, acquire an action instruction from the operator via the input device, interpret the acquired action instruction based on the determined task, and control the tele-operating system to perform an action using at least one tool of the plurality of tools based on the interpreted action instruction and the selected virtual tool representation, wherein the action approximates using a tool that corresponds to the selected virtual tool representation by using a different tool of the plurality of tools.

2 . The user interface according to claim 1 , wherein
the control circuit is configured to determine task parameters and task constraints based on the selected virtual tool representation, and interpret the acquired action instruction based on the determined task, task parameters and task constraints.

3 . The user interface according to claim 1 , wherein
each of the plural virtual tool representations comprises at least one adjustable tool constraint, wherein the adjustable tool constraint includes at least one of a direction constraint, a size constraint, a maximum speed setting, and a maximum thickness of removed material from an object.

4 . The user interface according to claim 3 , wherein
the plural virtual tool representations include elements that are configured to be manipulated by the operator for adapting the adjustable tool constraint.

5 . The user interface according to claim 3 , wherein
the plural virtual tool representations include operating elements, in particular a virtual slide control or a rotary control, that are configured to be manipulated by the operator for adapting numerically adjustable tool constraints.

6 . The user interface according to claim 1 , wherein
the user interface is part of a virtual reality system or augmented reality system.

7 . The user interface according to claim 1 , wherein
the input device includes a pointer device.

8 . A tele-operating system including a user interface, wherein the user interface comprises:
an output device configured to output plural virtual tool representations of a plurality of tools to an operator of the tele-operating system; an input device configured to receive a selection instruction from the operator for selecting one of the output virtual tool representations; a control circuit configured to determine a task on the selected virtual tool representation, control displaying the selected virtual tool representation to the operator via the output device, acquire an action instruction from the operator via the input device, interpret the acquired action instruction based on the determined task, and control the tele-operating system to perform an action using at least one tool of the plurality of tools based on the interpreted action instruction and the selected virtual tool representation, wherein the action approximates using a tool that corresponds to the selected virtual tool representation by using a different tool of the plurality of tools.

9 . The tele-operating system according to claim 8 , wherein the tele-operating system includes a multi-purpose physical tool configured to perform a plurality of actions corresponding to action capabilities of the plurality of tools.

10 . The tele-operating system according to claim 8 , wherein
the tele-operating system includes a specialized tool configured to perform actions corresponding to an action capability of the selected virtual tool representation, and a control circuit is configured to control the specialized tool based on task constraints of the selected virtual tool representation to perform the action.

11 . The tele-operating system according to claim 8 , wherein
the tele-operating system is configured to operate in a direct control mode, wherein in the direct control mode, the control circuit is configured to determine whether controlling the tele-operating system based on the interpreted action instruction for performing the action results in violating the at least one tool constraint, and, in case of determining violating at least one tool constraint, the control circuit is configured to output feedback information on the determined violation via the output device to the operator.

12 . The tele-operating system according to claim 8 , wherein
the tele-operating system is configured to operate in a direct control mode, wherein in the direct control mode, the tele-operating system is configured to determine whether controlling the tele-operating system based on the interpreted action instruction for performing the action results in approaching a violation of the at least one tool constraint.

13 . The tele-operating system according to claim 12 , wherein
the tele-operating system is configured to generate and output feedback information including a visual, acoustic or tactile warning when determining an approach to a violation of the at least one tool constraint.

14 . The tele-operating system according to claim 12 , wherein
the tele-operating system is configured to stop operating the tool or to decrease a speed of the tool when determining an approach to a violation of the at least one tool constraint.

15 . The tele-operating system according to claim 8 , wherein
the tele-operating system is configured to predict a tool trajectory of the tool and to project the predicted tool trajectory to constraints associated with the tool.

16 . The tele-operating system according to claim 8 , wherein
the tele-operating system is configured to operate in an indirect control mode, wherein in the indirect control mode, the tele-operating system is configured to determine and store in a data storage control commands for controlling the tele-operating system based on the interpreted action instruction for performing the action, and the tele-operating system is configured to obtain the stored control commands and to perform control of the tele-operating system based on the obtained control commands.

17 . The tele-operating system according to claim 16 , wherein
in the indirect control mode, the tele-operating system is configured to select the at least one tool for performing the action.

18 . The tele-operating system according to claim 16 , wherein
in the indirect control mode, the tele-operating system is configured to simulate an operation of the tele-operating system, in particular simulating and visualizing, via the output device, the simulated effects of operating the plurality of tools based on the stored control commands in real-time.

19 . A non-transitory computer readable medium having a computer readable program embodied therein, wherein the computer readable program, when executed on a computing device, causes the computing device to perform:
outputting, via an output device, plural virtual tool representations of a plurality of tools to an operator of the tele-operating system; obtaining, via an input device,

Assignees

• Honda Motor Co Ltd

Inventors

• Franzius Mathias
• Ruiken Dirk

Classifications

• G06F3/04847

  Interaction techniques to control parameter settings, e.g. interaction with sliders or dials · CPC title

• G06F3/038 Primary

  Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry · CPC title

• B25F5/00

  Details or components of portable power-driven tools not particularly related to the operations performed and not otherwise provided for · CPC title

• B25J9/1689

  Teleoperation · CPC title

• G06F3/04845

  for image manipulation, e.g. dragging, rotation, expansion or change of colour · CPC title