Virtual reality and augmented reality for industrial automation

Patent metadata

Field	Value
Publication number	US-10535202-B2
Application number	US-201916502491-A
Country	US
Kind code	B2
Filing date	Jul 3, 2019
Priority date	Nov 8, 2016
Publication date	Jan 14, 2020
Grant date	Jan 14, 2020

Abstract

An industrial visualization system generates and delivers virtual reality (VR) and augmented reality (AR) presentations of industrial facilities to wearable appliances to facilitate remote or enhanced interaction with automation systems within the facility. VR presentations can comprise three-dimensional (3D) holographic views of a plant facility or a location within a plant facility. The system can selectively render a scaled down view that renders the facility as a 3D scale model, or as a first-person view that renders the facility as a full-scale rendition that simulates the user's presence on the plant floor. Camera icons rendered in the VR presentation can be selected to switch to a live video stream generated by 360-degree cameras within the plant. The system can also render workflow presentations that guide users through the process of correcting detected maintenance issues.

First claim

What is claimed is:

1. A system, comprising:
a memory that stores executable components;
a processor, operatively coupled to the memory, that executes the executable components, the executable components comprising:
a device interface component configured to receive industrial data from industrial devices associated with an industrial facility, the industrial devices comprising at least an industrial controller that receives input data from at least one industrial input device and sends output data to one or more industrial output devices;
a client interface component configured to receive location data and orientation data from a first wearable appliance, wherein the location data and the orientation data indicate a current location and orientation, respectively, of the first wearable appliance; and
a rendering component configured to generate augmented reality presentation data that renders, on the first wearable appliance, an augmented reality presentation of the industrial facility based on the industrial data, the location data, and the orientation data,
wherein
the client interface component is further configured to receive, from the first wearable appliance, virtual note data defining an alphanumeric message, an indication of an industrial asset to which the alphanumeric message pertains, and a viewing permission criterion for the alphanumeric message; and
the rendering component is further configured to, in response to determining that a current location and a current orientation of a second wearable appliance causes the industrial asset to be within a field of view of the second wearable appliance, and that the second wearable appliance is associated with a user identity that satisfies the viewing permission criterion, render the alphanumeric message on the second wearable appliance.

2. The system of claim 1 , wherein the viewing permission criterion is at least one of an indication of a user role permitted to view the alphanumeric message or an identity of a specified person permitted to view the alphanumeric message.

3. The system of claim 1 , wherein the client interface component is configured to receive the virtual note data as spoken text submitted to the first wearable appliance.

4. The system of claim 1 , wherein the client interface component is configured to receive the indication of the industrial asset to which the alphanumeric message pertains as at least one of a gesture detected by the first wearable appliance or a verbal command submitted to the first wearable appliance.

5. The system of claim 1 , wherein the rendering component is configured to render the alphanumeric message on the second wearable appliance as an augmented reality overlay on or near the industrial asset.

6. The system of claim 1 , wherein the rendering component is configured to render a virtual note icon on the second wearable appliance as an augmented reality overlay on or near the industrial asset, and to render the alphanumeric message on the second wearable appliance in response to detecting a gesture or verbal command indicating selection of the virtual note icon.

7. The system of claim 1 , wherein
the client interface component is further configured to receive, from the first wearable appliance, an asset identifier of an industrial asset to be located, and
the rendering component is further configured to, in response to receipt of the asset identifier, render, on the first wearable appliance based on the location data and the orientation data, one or more graphical indicators that indicate a direction or location of the industrial asset.

8. The system of claim 1 , wherein
the client interface component is further configured to receive, from the first wearable appliance, a spoken request for a current status of a specified industrial asset, and
the rendering component is configured to, in response to receipt of the spoken request, retrieve a subset of the industrial data pertaining to the current status of the specified industrial asset and render, on the first wearable appliance, the subset of the industrial data.

9. The system of claim 1 , further comprising:
a video processing component configured to receive and store video data from video capture devices installed at respective locations in the industrial facility,
wherein the rendering component is further configured to
render, on the augmented reality presentation, camera icons at locations within the augmented reality presentation corresponding to the locations in the industrial facility at which the video capture devices are installed, and
in response to receipt of camera selection data from the first wearable appliance indicating that a camera icon, of the camera icons, has been selected via user input, transition the augmented reality presentation from a virtual view of the industrial facility to a video presentation that renders, on the first wearable device, a subset of the video data received from a video capture device, of the video capture devices, corresponding to the camera icon.

10. A method, comprising:
receiving, by a system comprising a processor, industrial data generated by industrial devices of an industrial facility, the industrial devices comprising at least an industrial controller that receives input data from at least one industrial input device and sends output data to one or more industrial output devices;
receiving, by the system, location data specifying a current location of a wearable appliance and orientation data specifying a current orientation of the wearable appliance;
generating, by the system, augmented reality presentation data that renders, on the wearable appliance, an augmented reality presentation based on the industrial data, the location data, and the orientation data;
receiving, by the system from the wearable appliance, virtual note data comprising an alphanumeric message, an indication of an industrial asset to which the alphanumeric message pertains, and a viewing permission criterion for the alphanumeric message; and
in response to determining that a current location and a current orientation of another wearable appliance places the industrial asset within a field of view of the other wearable appliance, and that the other wearable appliance is associated with a user identity that satisfies the viewing permission criterion, rendering, by the system, the alphanumeric message on the other wearable appliance.

11. The method of claim 10 , wherein the receiving the viewing permission criterion comprises receiving at least one of an indication of a user role permitted to view the alphanumeric message or an identity of a specified person permitted to view the alphanumeric message.

12. The method of claim 10 , wherein the receiving the virtual note data comprises receiving the alphanumeric message as spoken text submitted to the wearable appliance.

13. The method of claim 10 , wherein the receiving the indication of the industrial asset to which the alphanumeric message pertains comprises receiving the indication as at least one of a gesture detected by the wearable appliance or a verbal command submitted to the wearable appliance.

14. The method of claim 10 , wherein the rendering the alphanumeric message on the other wearable appliance comprises rendering the alphanumeric message as an augmented reality overlay positioned on or near the industrial asset within the field of view of the other wearable appliance.

15. The method of claim 10 , further comprising rendering, by the system, a virtual note icon as an augmented reality overlay on or near the industrial asset within the field of view of the other

Assignees

• Rockwell Automation Tech Inc

Inventors

• Schmirler Paul D
• Nguyen Thong T
• Nicoll Alex L
• Vasko David

Classifications

• G06T2219/024

  Multi-user, collaborative environment · CPC title

• G06F3/147

  using display panels · CPC title

• G06Q10/0633

  Workflow analysis · CPC title

• G06Q10/20

  Administration of product repair or maintenance · CPC title

• H04L63/08

  for authentication of entities (cryptographic mechanisms or cryptographic arrangements for entity authentication H04L9/32) · CPC title