Method for remote assistance and device

Patent metadata

Field	Value
Publication number	US-12346108-B2
Application number	US-202018041597-A
Country	US
Kind code	B2
Filing date	Aug 14, 2020
Priority date	Aug 14, 2020
Publication date	Jul 1, 2025
Grant date	Jul 1, 2025

Abstract

The present disclosure provides a method for remote assistance and device. 1. A method for remote assistance, the method comprising:determining a state of a machine deployed at a working site;adjusting a digital twin of the machine to be in the same state as the machine deployed at the working site;rendering the digital twin at a remote site using a remote rendering device to enable a remote person at the remote site to perform an operation on the digital twin; andmapping an effect of the operation rendered as visual assistance information onto the machine at the working site using a local rendering device deployed at the working site.

First claim

What is claimed is:

1. A method for remote assistance, the method comprising:
determining a state of a machine deployed at a working site;
setting a digital twin of the machine to the determined state, the digital twin including a virtual model of the machine on a server at a remote site;
rendering the digital twin at the remote site using a remote rendering device to enable a remote person at the remote site to perform an operation on the digital twin; and
mapping an effect of the operation by rendering the effect on the digital twin into visual assistance information displayed onto the machine at the working site using a local rendering device deployed at the working site.

2. The method according to claim 1 , wherein mapping an effect of the operation comprises:
identifying a component of the machine corresponding to a component of the digital twin operated on by the remote person;
generating visual content of the operation according to the effect caused by the operation on the digital twin; and
using the local rendering device to render the visual content at a position of the component of the machine.

3. The method according to claim 2 , wherein rendering the visual content at a position of the component of the machine comprises:
obtaining a working site image which depicts the working site and includes the machine;
generating an overlay image which includes the visual content at a position corresponding to the component in the working site image; and
using the local rendering device to render the overlay image superimposed onto the working site image.

4. The method according to claim 2 , wherein rendering the visual content at a position of the component of the machine comprises:
obtaining a position and a size of the component of the machine;
generating a projection image which includes the visual content with a position and a size corresponding to the position and the size of the component; and
using the local rendering device to project the projection image onto the machine.

5. The method according to claim 1 , further comprising rendering a second effect of a second operation on the machine at the remote site using a remote rendering device;
wherein the second operation is performed on the machine by a local person at the working site.

6. The method according to claim 2 , wherein the visual content comprises at least one of:
a digital mark made on a component of the machine;
moving pictures presenting a hand performing an operation; or
visual content presenting a component in a different state after an operation.

7. The method according to claim 1 , further comprising:
translating the operation into a control command; and
causing the machine to perform an action specified by the control command.

8. The method according to claim 1 , further comprising:
making information of a workpiece produced by the machine presented by the remote rendering device for the remote person to provide feedback; and
making feedback information made on the workpiece by the remote person presented by the local rendering device.

9. The method according to claim 1 , wherein setting the digital twin to the determined state comprises:
obtaining the information about the state of the machine from a device at the working site.

10. The method according to claim 9 , wherein obtaining the information about the state of the machine comprises at least one of:
receiving parameters of the machine from a control apparatus of the machine as the information; and
receiving sensor data from at least one sensor coupled to the machine as the information.

11. The method according to claim 9 , wherein setting the digital twin comprises modifying parameters of the digital twin until a simulated state of the digital twin is identical to the state specified by the information.

12. The method according to claim 1 , wherein rendering the effect on the digital twin comprises:
providing an image of the digital twin to the remote rendering device;
identifying in the digital twin a component being operated on by the remote person using a remote device which captures a gesture of the remote person; and
changing a visual effect of the component in the image rendered by the remote rendering device.

13. The method according to claim 1 , further comprising:
storing an assistance information entry into a database, the assistance information entry including at least one of: the operation performed by the remote person, data of the digital twin, the information about the state of the machine, information about a task of the machine obtained via a human-machine interface; information about a workpiece produced by the machine; and
in response to a request for assistance, rendering the visual assistance information specified in the assistance information entry onto a second machine using a second local rendering device deployed at a second working site of the second machine in response to a determination that the assistance information entry stored in the database matches with information in the request.

14. A computing device comprising:
a processor; and
a memory storing computer-readable instructions executable by the processor to:
determine a state of a machine deployed at a working site;
set a digital twin of the machine to the determined state, the digital twin including a virtual model of the machine on a server at a remote site;
render the digital twin at a remote site using a remote rendering device to enable a remote person at the remote site to perform an operation on the digital twin; and
render an effect of the operation as visual assistance information onto the machine at the working site using a local rendering device deployed at the working site.

15. A computer readable storage medium, storing computer-readable instructions executable by a processor to:
determine a state of a machine deployed at a working site;
set a digital twin of the machine to the determined state, the digital twin including a virtual model of the machine on a server at a remote site;
render the digital twin at a remote site using a remote rendering device to enable a remote person at the remote site to perform an operation on the digital twin; and
render an effect of the operation as visual assistance information onto the machine at the working site using a local rendering device deployed at the working site.

Assignees

• Siemens Ag

Inventors

• Roux Armin
• Zhang Bin
• Sun zhong yang
• Fan Shun Jie
• Jie Ming

Classifications

• G05D1/2245

  providing the operator with a purely computer-generated representation of the environment of the vehicle, e.g. virtual reality · CPC title

• G05D1/2247

  providing the operator with simple or augmented images from one or more cameras · CPC title

• G06F3/017

  Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883) · CPC title

• G06F3/011

  Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00) · CPC title

• G05D1/0038

  by providing the operator with simple or augmented images from one or more cameras located onboard the vehicle, e.g. tele-operation (images analyzed by a computer and used for automatic navigation G05D1/0246) · CPC title