Collaborative safety awareness for robotic applications enabled by augmented reality and other location-based technologies

Patent metadata

Field	Value
Publication number	US-12459124-B2
Application number	US-202318352547-A
Country	US
Kind code	B2
Filing date	Jul 14, 2023
Priority date	Jul 27, 2022
Publication date	Nov 4, 2025
Grant date	Nov 4, 2025

Abstract

A collaborative safety awareness system for robotic applications enabled by augmented reality (AR). One or more robots in a work cell are in communication with an application running on an AR device worn or held by an operator in the work cell. The AR device may be a headset apparatus, or a tablet or teach pendant device. A dynamic safety zone is created around the operator and the location of the dynamic safety zone is continuously updated by the robot controller based on the position of the AR device provided by the AR application, where the position of the AR device is determined using inertial sensors and visual odometry. The robot controller prohibits motion of the robot into the dynamic safety zone, and slows or reroutes the robot as needed to prevent an interference condition between the robot and the dynamic safety zone.

First claim

What is claimed is:

1 . A system for providing a dynamic safety zone for an operator in a machine work cell, said system comprising:
a machine operable in a work cell; a machine controller in communication with the machine, said controller including a processor and memory and being configured with machine operational control software; and an augmented reality (AR) device in communication with the machine controller, said AR device having one or more cameras, position tracking sensors and a display, said AR device further including a processor and memory configured to run an AR application; where the AR application establishes and continuously tracks a position and orientation of the AR device relative to a work cell coordinate frame, and continuously communicates the position and orientation of the AR device to the machine controller, and the machine controller establishes and moves a dynamic safety zone encompassing the operator based on the position and orientation of the AR device, and incorporates the dynamic safety zone in machine motion control computations, including preventing any part of the machine from entering the dynamic safety zone, where the dynamic safety zone has a size and behavior parameters definable by the operator in the AR application or the machine operational control software, where the behavior parameters include an offset of a center of the dynamic safety zone from the position of the AR device based on the orientation and type of the AR device, and an adjustment of the position and the size of the dynamic safety zone based on a velocity vector of the AR device.

2 . The system according to claim 1 wherein the AR device is a headset apparatus worn by the operator.

3 . The system according to claim 1 wherein the AR device is a smart phone, a tablet computing device or a robot teach pendant held by the operator.

4 . The system according to claim 1 wherein the AR application establishes the position and orientation of the AR device relative to the work cell coordinate frame by analyzing images of a visual marker having a known design and placed at a known location in the work cell coordinate frame.

5 . The system according to claim 1 wherein the position tracking sensors include one or more gyroscopes and one or more accelerometers, where the sensors provide signals to the processor in the AR device enabling continuous computation of changes in position and orientation of the AR device.

6 . The system according to claim 1 wherein the machine controller slows or reroutes motion of the machine in order to prevent a future interference of the machine with the dynamic safety zone.

7 . The system according to claim 1 wherein the AR application is used by the operator while the machine controller establishes and moves the dynamic safety zone based on the position and orientation of the AR device.

8 . The system according to claim 7 wherein the AR application provides virtual display elements superimposed on camera images of real-world objects, and the AR application is used by the operator for machine set-up, configuration, operation monitoring or troubleshooting.

9 . The system according to claim 1 wherein the machine is an industrial robot.

10 . The system according to claim 1 further comprising one or more additional machines in the work cell, one or more additional operators each having an AR device within the work cell, or both the additional machines and the additional operators, where a dynamic safety zone is defined for each of the operators and all of the dynamic safety zones are incorporated in the motion control computations for all of the machines to prevent any part of any machine from entering any of the dynamic safety zones.

11 . A method for providing a dynamic safety zone for an operator in a machine work cell, said machine work cell including at least one machine in communication with a machine controller, said method comprising:
establishing communication between an augmented reality (AR) device and the machine controller and starting an AR application on the AR device; establishing, by the AR application, a position and orientation of the AR device relative to a work cell coordinate frame and continuously tracking the position and orientation as the AR device is moved; continuously communicating the position and orientation of the AR device to the machine controller; creating and moving a dynamic safety zone encompassing the operator based on the position and orientation of the AR device, where the dynamic safety zone has a size and behavior parameters definable by the operator in the AR application or the machine operational control software, where the behavior parameters include an offset of a center of the dynamic safety zone from the position of the AR device based on the orientation and type of the AR device, and an adjustment of the position and the size of the dynamic safety zone based on a velocity vector of the AR device; and incorporating the dynamic safety zone in machine motion control computations, including preventing any part of the machine from entering the dynamic safety zone.

12 . The method according to claim 11 wherein the AR device is a headset apparatus worn by the operator, or the AR device is a smart phone, a tablet computing device or a robot teach pendant held by the operator.

13 . The method according to claim 11 wherein establishing the position and orientation of the AR device relative to the work cell coordinate frame includes analyzing images from a camera in the AR device of a visual marker having a known design and placed at a known location in the work cell coordinate frame, and continuously tracking the position and orientation includes analyzing signals from position tracking sensors in the AR device.

14 . The method according to claim 11 wherein the machine controller slows or reroutes motion of the machine in order to prevent a future interference of the machine with the dynamic safety zone.

15 . The method according to claim 11 wherein the AR application is used by the operator while the machine controller establishes and moves the dynamic safety zone based on the position and orientation of the AR device.

16 . The method according to claim 11 wherein the AR application provides virtual display elements superimposed on camera images of real-world objects, and the AR application is used by the operator for machine set-up, configuration, operation monitoring or troubleshooting.

17 . The method according to claim 11 wherein the at least one machine is an industrial robot.

18 . A method for providing a dynamic safety zone for an operator in a machine work cell, said method comprising carrying a location-tracking device by the operator, continuously wirelessly communicating a position of the location-tracking device to a machine controller, defining a dynamic safety zone encompassing the operator based on the position of the location-tracking device, and incorporating the dynamic safety zone in machine motion control computations by the machine controller, including preventing any part of the machine from entering the dynamic safety zone, where the dynamic safety zone has a size and behavior parameters definable by the operator in the AR application or the machine operational control software, where the behavior parameters include an offset of a center of the dynamic safety zone from the position of the AR device based on the orientation and type of the AR device, and an adjustment of the position and the size of the dynamic safety zone base

Assignees

• Fanuc America Corp

Inventors

• Jung Derek

Classifications

• G05B2219/39449

  Pendant, pda displaying camera images overlayed with graphics, augmented reality · CPC title

• G05B2219/40202

  Human robot coexistence · CPC title

• G05B2219/32014

  Augmented reality assists operator in maintenance, repair, programming, assembly, use of head mounted display with 2-D 3-D display and voice feedback, voice and gesture command · CPC title

• B25J13/06

  Control stands, e.g. consoles, switchboards · CPC title

• G06T7/73

  using feature-based methods · CPC title