Virtual reality and augmented reality for industrial automation

Patent metadata

Field	Value
Publication number	US-11159771-B2
Application number	US-202016915194-A
Country	US
Kind code	B2
Filing date	Jun 29, 2020
Priority date	Nov 8, 2016
Publication date	Oct 26, 2021
Grant date	Oct 26, 2021

Abstract

An industrial visualization system generates and delivers virtual reality (VR) and augmented reality (AR) presentations of industrial facilities to wearable appliances to facilitate remote or enhanced interaction with automation systems within the facility. VR presentations can comprise three-dimensional (3D) holographic views of a plant facility or a location within a plant facility. The system can selectively render a scaled down view that renders the facility as a 3D scale model, or as a first-person view that renders the facility as a full-scale rendition that simulates the user's presence on the plant floor. Camera icons rendered in the VR presentation can be selected to switch to a live video stream generated by 360-degree cameras within the plant. The system can also render workflow presentations that guide users through the process of correcting detected maintenance issues.

First claim

What is claimed is:

1. A system, comprising:
a processor, operatively coupled to a memory, that executes executable components stored on the memory, the executable components comprising:
a rendering component configured to generate an augmented reality presentation of an industrial facility on a client device based on industrial data received from industrial devices of the industrial facility and a plant model that models a physical layout of industrial assets within the industrial facility and behaviors of the industrial assets;
a simulation component configured to perform simulated control of the industrial assets by an industrial control program based on emulated execution of the industrial control program and exchange of virtualized input and output data between the industrial control program and the plant model, and to determine at least one of a performance metric or a financial metric for the industrial assets based the simulated control; and
a monitoring component configured to identify occurrence of a maintenance issue based on a determination that a subset of the industrial data satisfies a condition indicative of the maintenance issue,
wherein the rendering component is further configured to, in response to identification of the maintenance issue, retrieve workflow data defining a workflow for correcting the maintenance issue, and render the workflow as part of the augmented reality presentation.

2. The system of claim 1 , wherein the plant model defines graphical representations of the industrial assets and animation properties for the graphical representations.

3. The system of claim 1 , wherein the behaviors modeled by the plant model comprise at least one of a movement, a speed, a temperature, a flow, or a fill level associated with one or more of the industrial assets in response to values of the virtualized output data generated by the simulation component.

4. The system of claim 1 , wherein the simulation component is further configured to perform predictive analytics for the industrial assets based on the simulated control.

5. The system of claim 4 , wherein the simulation component is further configured to generate at least one of a maintenance schedule for one or more of the industrial assets or a replacement schedule for one or more of the industrial assets based on a result of the predictive analysis.

6. The system of claim 1 , wherein
the workflow data defines a sequence of steps to be performed to correct the maintenance issue, and
the rendering component is configured to, for a current step of the sequence of steps, render graphical indications on the client device that provide instruction regarding performance of the current step.

7. The system of claim 1 wherein the maintenance issue is at least one of a deviation of a key performance indicator from a defined range, detection of a machine failure or a device failure, or an alarm condition of a machine or one of the industrial devices.

8. The system of claim 1 , further comprising a video processor configured to store video data received from video capture devices installed at respective locations of the industrial facility,
wherein
the augmented reality presentation comprises camera icons rendered at locations within the augmented reality presentation corresponding to the locations of the industrial facility at which the video capture devices are installed, and
the rendering component is configured to, in response to receipt of camera selection data from the client device indicating that a camera icon of the camera icons has been selected via user input, transition the augmented reality presentation from a virtual view of the industrial facility to a video presentation that renders, on the client device, a subset of the video data received from a video capture device, of the video capture devices, corresponding to the camera icon.

9. A method, comprising:
rendering, by a system comprising a processor, an augmented reality representation of an industrial facility on a client device based on industrial data received from industrial devices associated with the industrial facility and a plant model that models a physical layout of industrial assets within the industrial facility and behaviors of the industrial assets;
simulating, by the system, control of the industrial assets by an industrial control program based on emulated execution of the industrial control program and exchange of virtualized input and output data between the industrial control program and the plant model;
determining, by the system based on a result of the simulating, at least one of a performance metric or a financial metric for the industrial assets;
identifying, by the system, occurrence of a maintenance issue based on a determination that a subset of the industrial data satisfies a condition indicative of the maintenance issue; and
in response to the identifying of the maintenance issue:
retrieving workflow data defining a workflow for correcting the maintenance issue, and
rendering the workflow as part of the augmented reality representation.

10. The method of claim 9 , wherein the plant model defines graphical representations of the industrial assets and animation properties for the graphical representations.

11. The method of claim 9 , wherein the behaviors modeled by the plant model comprise at least one of a movement, a speed, a temperature, a flow, or a fill level associated with one or more of the industrial assets in response to values of the virtualized output data generated by the simulation component.

12. The method of claim 9 , further comprising performing, by the system, predictive analytics for the industrial assets based on another result of the simulating.

13. The method of claim 12 , further comprising generating, by the system, at least one of a maintenance schedule for one or more of the industrial assets or a replacement schedule for one or more of the industrial assets based on the predictive analysis.

14. The method of claim 9 , wherein the workflow data defines a sequence of steps to be performed to correct the maintenance issue, and
the rendering the workflow comprises, for a current step of the sequence of steps, rendering graphical indications on the client device that provide instruction regarding performance of the current step.

15. The method of claim 9 , wherein the maintenance issue is at least one of a deviation of a key performance indicator from a defined range, detection of a machine failure or a device failure, or an alarm condition of a machine or one of the industrial devices.

16. The method of claim 9 , wherein the rendering the augmented reality representation comprises rendering camera icons at locations on the augmented reality representation corresponding to locations of video capture devices within the industrial facility, and
the method further comprises:
in response to receipt of camera selection data from the client device identifying a camera icon, of the camera icons, that has been selected via user input, transitioning, by the system, the augmented reality representation from a virtual view of the industrial facility to a video presentation; and
streaming, by the system to the client device via the video presentation, a subset of the video data received from a video capture device, of the video capture devices, corresponding to the camera icon.

17. A non-transitory computer-readable medium having stored thereon instructions that, in response to execution, cause a system comprising a processor to perform operations, the operations comprising:
rendering an

Assignees

• Rockwell Automation Tech Inc

Inventors

• Schmirler Paul D
• Nguyen Thong T
• Nicoll Alex L
• Vasko David

Classifications

• H04N23/698

  for achieving an enlarged field of view, e.g. panoramic image capture · CPC title

• H04N23/90

  Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums · CPC title

• H04N13/398

  Synchronisation thereof; Control thereof · CPC title

• H04N13/366

  using viewer tracking · CPC title

• H04N13/332

  Displays for viewing with the aid of special glasses or head-mounted displays [HMD] · CPC title