Remote operation device and remote operation system

Patent metadata

Field	Value
Publication number	US-2023003002-A1
Application number	US-202017782756-A
Country	US
Kind code	A1
Filing date	Nov 30, 2020
Priority date	Dec 19, 2019
Publication date	Jan 5, 2023
Grant date	—

Abstract

A remote operation device and a remote operation system that can reduce or eliminate deviation of the position of an indicator image of a working unit such as a line image in an image with respect to the position of the working unit in a real space. In some case, because of a cause such as a communication failure, a position posture coordinate value of a working unit corresponding to captured image data in a work environment image deviates from a position posture coordinate value of the working unit estimated according to an operation form of a remote operation mechanism. In this case, a semitransparent indicator image Img2(τ2) simulating a bucket is superimposed on a work environment image Img1(τ2) in a position where the bucket is estimated to be originally present and deviating from an image region representing the bucket.

First claim

1 . A remote operation device for remotely operating a working machine including a working mechanism, the remote operation device comprising:
remote wireless communication equipment having a communication function with real machine wireless communication equipment mounted on the working machine; a remote image output device; a remote operation mechanism; a remote control element that recognizes an operation form of the remote operation mechanism by an operator and causes the remote wireless communication equipment to transmit a command signal corresponding to the operation form to the real machine wireless communication equipment; a first output control element that causes the remote image output device to output a work environment image including a working unit of the working mechanism acquired through an imaging device mounted on the working machine, transmitted from the real machine wireless communication equipment, and received by the remote wireless communication equipment; and a second output control element that estimates, based on an operation form of the remote operation mechanism at a first point in time recognized by the remote control element, a space occupying form of the working unit at a second point in time later than the first point in time, superimposes, on the work environment image, at a third point in time included in a period from the first point in time to the second point in time, an indicator image representing the estimated space occupying form of the working unit at the second point in time without spoiling visibility of the working unit, and causes the remote image output device to output the indicator image.

2 . The remote operation device according to claim 1 , wherein the second output control element superimposes the indicator image on the work environment image at the third point in time, which is an intermediate point in time between the first point in time and the second point in time, and causes the remote image output device to output the indicator image.

3 . The remote operation device according to claim 2 , wherein
the second output control element estimates the space occupying form of the working unit with a point in time later than the first point in time by a total delay time set as the second point in time, the total delay time being a total of a first delay time, which is at least one of a first communication delay time, which is a communication delay time from when a command signal corresponding to an operation form of the remote operation mechanism is transmitted from the remote wireless communication equipment until when the command signal is received by the real machine wireless communication equipment, and a first response delay time, which is a response delay time from when the command signal is received by the real machine wireless communication equipment until when operation of the working machine is controlled according to the command signal, and a second delay time, which is at least one of a second communication delay time, which is a communication delay time from when the work environment image is transmitted from the real machine wireless communication equipment until when the work environment image is received by the remote wireless communication equipment, and a second response delay time, which is a response delay time from when the work environment image is received by the remote wireless communication equipment until when the work environment image is output to the remote image output device, and the second output control element superimposes the indicator image on the work environment image at the third point in time, which is a point in time later than the first point in time by the first delay time, and causes the remote image output device to output the indicator image.

4 . The remote operation device according to claim 3 , wherein the second output control element superimposes the indicator image on the work environment image at the third point in time, which is a point in time later than the first point in time by the first response delay time serving as the first delay time, and causes the remote image output device to output the indicator image.

5 . The remote operation device according to claim 1 , wherein the second output control element superimposes the indicator image on the work environment image at the third point in time, which is a same point in time as the second point in time, and causes the remote image output device to output the indicator image.

6 . The remote operation device according to claim 5 , wherein the second output control element estimates the space occupying form of the working unit with a point in time later than the first point in time by a total delay time set as the second point in time, the total delay time being a total of
a first delay time, which is at least one of a first communication delay time, which is a communication delay time from when a command signal corresponding to an operation form of the remote operation mechanism is transmitted from the remote wireless communication equipment until when the command signal is received by the real machine wireless communication equipment, and a first response delay time, which is a response delay time from when the command signal is received by the real machine wireless communication equipment until when operation of the working machine is controlled according to the command signal, and a second delay time, which is at least one of a second communication delay time, which is a communication delay time from when the work environment image is transmitted from the real machine wireless communication equipment until when the work environment image is received by the remote wireless communication equipment, and a second response delay time, which is a response delay time from when the work environment image is received by the remote wireless communication equipment until when the work environment image is output to the remote image output device.

7 . The remote operation device according to claim 1 , wherein
the remote control element recognizes an operating form of the working mechanism detected through a state sensor mounted on the working machine, transmitted from the real machine wireless communication equipment, and received by the remote wireless communication equipment, and the second output control element estimates a space occupying form of the working unit at the second point in time based on a deviation between an operation form of the remote operation mechanism at the first point in time recognized by the remote control element and an operating form of the working mechanism.

8 . The remote operation device according to claim 1 , wherein, when a stop of first designated operation is recognized as an operation form of the remote operation mechanism by the remote control element, the second output control element stops an output of the indicator image in the remote image output device.

9 . The remote operation device according to claim 1 , wherein, when a start of second designated operation is recognized as an operation form of the remote operation mechanism by the remote control element, the second output control element causes the remote image output device to temporarily output the indicator image.

10 . A remote operation system configured by the remote operation device according to claim 1 and the working machine.

Assignees

• Kobelco Constr Mach Co Ltd
• Univ Hiroshima

Inventors

• Ito Masaru
• Kurita Yuichi

Classifications

• G08C17/02

  using a radio link · CPC title

• E02F9/264

  Sensors and their calibration for indicating the position of the work tool · CPC title

• E02F3/435

  for dipper-arms, backhoes or the like · CPC title

• H04N7/181

  for receiving images from a plurality of remote sources · CPC title

• E02F9/205 Primary

  Remotely operated machines, e.g. unmanned vehicles (E02F3/8866 takes precedence) · CPC title