Virtual reality and augmented reality for industrial automation

Patent metadata

Field	Value
Publication number	US-10388075-B2
Application number	US-201715718907-A
Country	US
Kind code	B2
Filing date	Sep 28, 2017
Priority date	Nov 8, 2016
Publication date	Aug 20, 2019
Grant date	Aug 20, 2019

Abstract

An industrial visualization system generates and delivers virtual reality (VR) and augmented reality (AR) presentations of industrial facilities to wearable appliances to facilitate remote or enhanced interaction with automation systems within the facility. VR presentations can comprise three-dimensional (3D) holographic views of a plant facility or a location within a plant facility. The system can selectively render a scaled down view that renders the facility as a 3D scale model, or as a first-person view that renders the facility as a full-scale rendition that simulates the user's presence on the plant floor. Camera icons rendered in the VR presentation can be selected to switch to a live video stream generated by 360-degree cameras within the plant. The system can also render workflow presentations that guide users through the process of correcting detected maintenance issues.

First claim

What is claimed is:

1. A system, comprising:
a memory that stores executable components;
a processor, operatively coupled to the memory, that executes the executable components, the executable components comprising:
a device interface component configured to receive industrial data from industrial devices associated with an industrial facility, the industrial devices comprising at least an industrial controller that receives input data from at least one industrial input device and sends output data to one or more industrial output devices;
a client interface component configured to receive identity data, location data, and orientation data from a wearable appliance, wherein the identity data defines a user identity associated with the wearable appliance, and the location data and the orientation data indicate a current location and orientation, respectively, of the wearable appliance;
a monitoring component configured to identify occurrence of a maintenance issue based on a determination that a subset of the industrial data satisfies a condition indicative of the maintenance issue; and
a rendering component configured to, in response to identification of the maintenance issue, retrieve workflow data defining a workflow for correcting the maintenance issue, and generate augmented reality presentation data that renders, on the wearable appliance, an augmented reality presentation based on the workflow data, the identity data, the location data, and the orientation data,
wherein the client interface component is further configured to receive, from the wearable appliance, virtual note data defining an alphanumeric message, an indication of an industrial asset to which the alphanumeric message pertains, and a viewing permission criterion for the alphanumeric message, and
the rendering component is further configured to, in response to determining that a current location and a current orientation of a second wearable appliance causes the industrial asset to be within a field of view of the second wearable appliance, and that the second wearable appliance is associated with a user identity that satisfies the viewing permission criterion, render the alphanumeric message on the second wearable appliance.

2. The system of claim 1 , wherein the rendering component is configured to select the wearable appliance for delivery of the augmented reality presentation data based on a correlation between the identity data and at least one of a type of the maintenance issue or a location of the maintenance issue.

3. The system of claim 1 , wherein
the workflow data defines a sequence of steps to be performed to correct the maintenance issue, and
the rendering component is configured to, for a current step of the sequence of steps, render graphical indications on the wearable appliance that provide instruction regarding performance of the current step.

4. The system of claim 3 , wherein the rendering component is configured to render the graphical indications at a location within a display area of the wearable appliance selected based on the location data and the orientation data.

5. The system of claim 3 , wherein the graphical indications comprise at least one of an alphanumeric instruction, a graphical arrow rendered on a display screen of the wearable appliance indicating a direction toward the maintenance issue, a graphical indicator rendered at a position of the display screen that identifies an industrial device or a machine component on which the current step is to be performed, an indication that the current step has been correctly performed, or an indication that the current step has been incorrectly performed.

6. The system of claim 3 , wherein the rendering component is further configured to render a video on the wearable appliance demonstrating performance of the current step.

7. The system of claim 1 , wherein the rendering component is configured to select the workflow data from a library of defined workflows stored on the memory based on the maintenance issue.

8. The system of claim 1 , wherein the maintenance issue is at least one of a deviation of a key performance indicator from a defined range, detection of a machine failure or a device failure, or an alarm condition of a machine or one of the industrial devices.

9. The system of claim 3 , wherein
the monitoring component is further configured to detect, based on monitoring of at least one of the industrial data, the location data, or the orientation data, a deviation from the workflow, and
the rendering component is further configured to, in response to detection by the monitoring component of the deviation from the workflow, render a corrective instruction on the wearable device.

10. The system of claim 9 , wherein the monitoring component is further configured to calculate a performance metric based on at least one of a number of deviations from the workflow or a speed of completion of the workflow, and to store the performance metric in association with the user identity.

11. The system of claim 1 , wherein
the client interface component is further configured to receive, from the wearable appliance, an asset identifier of an industrial asset to be located, and
the rendering component is further configured to, in response to receipt of the asset identifier, render, on the wearable appliance based on the location data and the orientation data, one or more graphical indicators that indicate a direction or location of the industrial asset.

12. The system of claim 1 , wherein
the client interface component is further configured to receive, from the wearable appliance, a spoken request for a current status of a specified industrial asset, and
the rendering component is configured to, in response to receipt of the spoken request, retrieve a subset of the industrial data pertaining to the current status of the specified asset and render, on the wearable appliance, the subset of the industrial data.

13. A method, comprising:
receiving, by a system comprising a processor, industrial data generated by industrial devices of an industrial facility, the industrial devices comprising at least an industrial controller that receives input data from at least one industrial input device and sends output data to one or more industrial output devices;
receiving, by the system, identity data specifying a user identity associated with a wearable appliance, location data specifying a current location of the wearable appliance, and orientation data specifying a current orientation of the wearable appliance;
identifying, by the system, occurrence of a maintenance issue based on a determination that a subset of the industrial data satisfies a condition indicative of the maintenance issue;
in response to the identifying of the maintenance issue, retrieving, by the system, workflow data that defines a workflow for correcting the maintenance issue;
generating, by the system, augmented reality presentation data that renders, on the wearable appliance, an augmented reality presentation based on the workflow data, the identity data, the location data, and the orientation data;
receiving, by the system from the wearable appliance, virtual note data comprising an alphanumeric message, an indication of an industrial asset to which the alphanumeric message pertains, and a viewing permission criterion for the alphanumeric message; and
in response to determining that a current location and a current orientation of another wearable appliance places the industrial asset within a field of view of the other wearable appliance, and that the other wearable appliance is associated with a user identity that satisfies the viewing per

Assignees

• Rockwell Automation Tech Inc

Inventors

• Schmirler Paul D
• Nguyen Thong T
• Nicoll Alex L
• Vasko David

Classifications

• G05B2219/32014

  Augmented reality assists operator in maintenance, repair, programming, assembly, use of head mounted display with 2-D 3-D display and voice feedback, voice and gesture command · CPC title

• G05B23/0216

  Human interface functionality, e.g. monitoring system providing help to the user in the selection of tests or in its configuration · CPC title

• G06Q10/20

  Administration of product repair or maintenance · CPC title

• G06F3/147

  using display panels · CPC title

• G06T19/006 Primary

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title