Teleoperation system and method thereof

Patent metadata

Field	Value
Publication number	US-2025042033-A1
Application number	US-202318361918-A
Country	US
Kind code	A1
Filing date	Jul 31, 2023
Priority date	Jul 31, 2023
Publication date	Feb 6, 2025
Grant date	—

Abstract

A method and teleoperation system, includes provision of first information on an environment of a teleoperated device by at least one sensor module that is included in the teleoperated device and prediction of an action intended by the operator based on an input of the operator by an action predictor module that is included in the information gathering assistant module. It is determined whether first information on an environment is sufficient to execute the intended action in the environment by a reasoning agent module. Then second information on the environment is determined, which is suitable to enhance the environment information which is useful to execute the intended action in the environment by the reasoning agent module. Provision of second information is controlled by at least one information provision unit, and the first and second information on the environment is fused by an information fusion module.

First claim

What is claimed is:

1 . A teleoperation system, comprising:
a teleoperated device; an information gathering assistant module; at least one information provision unit; an information fusion module; and a user interface, wherein the teleoperated device comprises at least one sensor module that is configured to provide first information on an environment of the teleoperated system, the user interface comprises an input device, the information gathering assistant module comprises an action predictor module configured to predict an intended action based on an input of the operator and a reasoning agent module configured to determine second information on the environment, and is configured to control provision of the second information on the environment by at least one information provision unit, and the information fusion module is configured to fuse the first and second information.

2 . The teleoperation system according to claim 1 , wherein
the information gathering assistant module comprises a decision engine module that is configured to determine at least one information provision unit that is capable of providing second information.

3 . The teleoperation system according to claim 1 , wherein
the information gathering assistant module comprises a decision engine module that is configured to determine the type and/or amount of the at least one information provision unit for providing second information based on at least one of a current state of the system, operator preferences and success rates estimated by the reasoning agent.

4 . The teleoperation system according to claim 1 , wherein
the at least one information provision unit comprises an electrical storage, and the teleoperation system and/or the teleoperated device comprises a base- or charging station to charge the at least one information provision unit.

5 . The teleoperation system according to claim 1 , wherein
first and second information is fused into a 3D visual representation.

6 . The teleoperation system according to claim 1 , wherein
the user interface comprises a visual interface that is configured to visualize the fused information.

7 . The teleoperation system according to claim 1 , wherein
the reasoning agent module is further configured to determine viewpoints in the environment that define the location at which at least one information provision unit can be positioned in order to provide second information on the environment and/or to determine a region of interest.

8 . The teleoperation system according to claim 7 , wherein
the region of interest is determined based on at least one of the inputs of the operator, an assistant function behavior and a determination of a low-quality region.

9 . The teleoperation system according to claim 7 , wherein
the at least one information provision unit is removably attachable to a stationary platform at a position that coincides with any of the determined viewpoints.

10 . The teleoperation system according to claim 1 , wherein
the teleoperation system further comprises a persistent memory that is configured to store the fused information on the environment and/or store a past environment information.

11 . The teleoperation system according to claim 1 , wherein
at least one of the information gathering assistant module and the information provision unit comprises an information provision unit controller that is configured to provide control signals based on which the information provision unit provides second information on the environment.

12 . The teleoperation system according to claim 1 , wherein
the at least one information provision unit comprises at least one of a sensor and a mirror.

13 . The teleoperation system according to claim 11 , wherein
the information provision unit controller is configured to change a pose of at least one of the sensor and the mirror.

14 . The teleoperation system according to claim 1 , wherein
the at least one information provision unit comprises a movable platform and/or a controllable movable platform, and/or is removably attachable to a movable and/or a controllable movable platform.

15 . The teleoperation system according to claim 14 , wherein
the controllable movable platform is a portion of the teleoperated device.

16 . The teleoperation system according to claim 15 , wherein
the portion of the teleoperated device belongs to a portion that is controllable without changing the outcome of the intended action of the operator.

17 . The teleoperation system according to claim 15 , wherein
in case that the portion of the teleoperated device belongs to a portion that is not controllable without changing the outcome of the intended action of the operator, the information gathering assistant module comprises a decision agent module that is configured to determine the at least one information provision unit that is attached to the portion of the teleoperated device as being capable of providing second information on the environment only if the movable portion is currently not controlled by the operator.

18 . The teleoperation system according to claim 14 , wherein
the information provision unit controller is further configured to calculate a trajectory for a respective controllable movable platform.

19 . The teleoperation system according to claim 18 , wherein
the calculated trajectory is updated to avoid collisions with the teleoperated device and the at least one information provision unit.

20 . A method for an teleoperation system, comprising:
Providing first information on an environment of a teleoperated device by at least one sensor module that is comprised in the teleoperated device; predicting an intended action based on an input of the operator by an action predictor module that is comprised in the information gathering assistant module; determining if the first information on an environment is sufficient to execute the intended action in the environment by a reasoning agent module; determining second information on the environment suitable to enhance the environment information which is useful to execute the intended action in the environment by the reasoning agent module; controlling provision of second information by at least one information provision unit by the information gathering assistant module; and fusing first and second information on the environment by an information fusion module.

Assignees

• Honda Res Inst Europe Gmbh

Inventors

• Schön Oliver
• Ruiken Dirk
• Franzius Mathias
• Weigel Martin

Classifications

• G05B19/4155

  characterised by program execution, i.e. part program or machine function execution, e.g. selection of a program · CPC title

• G05B2219/40174

  Robot teleoperation through internet · CPC title

• B25J9/1689 Primary

  Teleoperation · CPC title