System and method for using virtual/augmented reality for interaction with collaborative robots in manufacturing or industrial environment

Patent metadata

Field	Value
Publication number	US-2021237275-A1
Application number	US-202016836569-A
Country	US
Kind code	A1
Filing date	Mar 31, 2020
Priority date	Jan 30, 2020
Publication date	Aug 5, 2021
Grant date	—

Abstract

A method includes determining a movement of an industrial robot in a manufacturing environment from a first position to a second position. The method also includes displaying an image showing a trajectory of the movement of the robot on a wearable headset. The displaying of the image comprises at least one of: displaying an augmented reality (AR) graphical image or video of the trajectory superimposed on a real-time actual image of the robot, or displaying a virtual reality (VR) graphical image or video showing a graphical representation of the robot together with the trajectory.

First claim

What is claimed is:

1 . A method comprising:
determining a movement of an industrial robot in a manufacturing environment from a first position to a second position; and displaying an image showing a trajectory of the movement of the robot on a wearable headset, comprising at least one of:
displaying an augmented reality (AR) graphical image or video of the trajectory superimposed on a real-time actual image of the robot; or
displaying a virtual reality (VR) graphical image or video showing a graphical representation of the robot together with the trajectory.

2 . The method of claim 1 , further comprising:
receiving a user input indicating a desired second movement of the robot; providing instructions to the robot to perform the second movement; and displaying a second image on the wearable headset showing a trajectory of the second movement of the robot.

3 . The method of claim 1 , wherein the trajectory of the movement of the robot is displayed as the robot moves.

4 . The method of claim 1 , further comprising:
changing the image showing the trajectory of the movement of the robot in response to a change of location or orientation, relative to the robot, of a person wearing the wearable headset.

5 . The method of claim 1 , further comprising:
displaying work instructions of a manufacturing task on the wearable headset, the work instructions related to movement of the robot during the manufacturing task.

6 . The method of claim 1 , further comprising:
receiving, from at least one sensor, location information of at least one of the robot or a person wearing the wearable headset; and controlling the movement of the robot based on the location information to maintain separation between the robot and the person.

7 . The method of claim 1 , further comprising:
transmitting information from the wearable headset to a workstation in the manufacturing environment for path programming of the robot, the information related to one or more actions of a person wearing the wearable headset during a training simulation involving the robot.

8 . A system comprising:
at least one memory configured to store instructions; and at least one processor coupled to the at least one memory and configured when executing the instructions to:
determine a movement of an industrial robot in a manufacturing environment from a first position to a second position; and
control a wearable headset to display an image showing a trajectory of the movement of the robot, comprising at least one of:
control the wearable headset to display an augmented reality (AR) graphical image or video of the trajectory superimposed on a real-time actual image of the robot; or
control the wearable headset to display a virtual reality (VR) graphical image or video showing a graphical representation of the robot together with the trajectory.

9 . The system of claim 8 , wherein the at least one processor is further configured to:
receive a user input indicating a desired second movement of the robot; provide instructions to the robot to perform the second movement; and control the wearable headset to display a second image showing a trajectory of the second movement of the robot.

10 . The system of claim 8 , wherein the at least one processor is configured to control the wearable headset to display the trajectory of the movement of the robot as the robot moves.

11 . The system of claim 8 , wherein the at least one processor is further configured to:
control the wearable headset to change the image showing the trajectory of the movement of the robot in response to a change of location or orientation, relative to the robot, of a person wearing the wearable headset.

12 . The system of claim 8 , wherein the at least one processor is further configured to:
control the wearable headset to display work instructions of a manufacturing task, the work instructions related to movement of the robot during the manufacturing task.

13 . The system of claim 8 , wherein the at least one processor is further configured to:
receive, from at least one sensor, location information of at least one of the robot or a person wearing the wearable headset; and control movement of the robot based on the location information to maintain separation between the robot and the person.

14 . The system of claim 8 , wherein the at least one processor is further configured to:
transmit information from the wearable headset to a workstation in the manufacturing environment for path programming of the robot, the information related to one or more actions of a person wearing the wearable headset during a training simulation involving the robot.

15 . A non-transitory computer readable medium containing instructions that when executed cause at least one processor to:
determine a movement of an industrial robot in a manufacturing environment from a first position to a second position; and control a wearable headset to display an image showing a trajectory of the movement of the robot on the wearable headset, comprising at least one of:
control the wearable headset to display an augmented reality (AR) graphical image or video of the trajectory superimposed on a real-time actual image of the robot; or
control the wearable headset to display a virtual reality (VR) graphical image or video showing a graphical representation of the robot together with the trajectory.

16 . The non-transitory computer readable medium of claim 15 , wherein the instructions further cause the at least one processor to:
receive a user input indicating a desired second movement of the robot; provide instructions to the robot to perform the second movement; and control the wearable headset to display a second image showing a trajectory of the second movement of the robot.

17 . The non-transitory computer readable medium of claim 15 , wherein the trajectory of the movement of the robot is displayed as the robot moves.

18 . The non-transitory computer readable medium of claim 15 , wherein the instructions further cause the at least one processor to:
control the wearable headset to change the image showing the trajectory of the movement of the robot in response to a change of location or orientation, relative to the robot, of a person wearing the wearable headset.

19 . The non-transitory computer readable medium of claim 15 , wherein the instructions further cause the at least one processor to:
control the wearable headset to display work instructions of a manufacturing task, the work instructions related to movement of the robot during the manufacturing task.

20 . The non-transitory computer readable medium of claim 15 , wherein the instructions further cause the at least one processor to:
receive, from at least one sensor, location information of at least one of the robot or a person wearing the wearable headset; and control movement of the robot based on the location information to maintain separation between the robot and the person.

Assignees

• Raytheon Co

Inventors

• Stone Kristen M

Classifications

• G06T19/006

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G06T11/60

  Creating or editing images; Combining images with text · CPC title

• G06T7/70

  Determining position or orientation of objects or cameras (camera calibration G06T7/80) · CPC title

• G06T7/20

  Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51) · CPC title

• G06F3/011

  Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00) · CPC title