Virtual reality and augmented reality for industrial automation

Patent metadata

Field	Value
Publication number	US-11265513-B2
Application number	US-202016915152-A
Country	US
Kind code	B2
Filing date	Jun 29, 2020
Priority date	Nov 8, 2016
Publication date	Mar 1, 2022
Grant date	Mar 1, 2022

Abstract

An industrial visualization system generates and delivers virtual reality (VR) and augmented reality (AR) presentations of industrial facilities to wearable appliances to facilitate remote or enhanced interaction with automation systems within the facility. VR presentations can comprise three-dimensional (3D) holographic views of a plant facility or a location within a plant facility. The system can selectively render a scaled down view that renders the facility as a 3D scale model, or as a first-person view that renders the facility as a full-scale rendition that simulates the user's presence on the plant floor. Camera icons rendered in the VR presentation can be selected to switch to a live video stream generated by 360-degree cameras within the plant. The system can also render workflow presentations that guide users through the process of correcting detected maintenance issues.

First claim

What is claimed is:

1. A system, comprising:
a processor, operatively coupled to a memory, that executes executable components stored on the memory, the executable components comprising:
a rendering component configured to generate, on a first client device, an augmented reality presentation of a portion of an industrial system located within a field of view of the first client device; and
a monitoring component configured to identify occurrence of a maintenance issue based on a determination that industrial data from one or more industrial devices of the industrial system satisfies a condition indicative of the maintenance issue; and
a client interface component configured to receive, from the first client device, a virtual note, an indication of an industrial asset of the industrial system to which the virtual note pertains, and a viewing permission criterion for the virtual note,
wherein the rendering component is further configured to,
in response to identification of the maintenance issue, retrieve workflow data defining a workflow for correcting the maintenance issue, and render, on the first client device, the workflow data as part of the augmented reality presentation, and
in response to determining that the industrial asset is being presented on a second client device, and that the second client device satisfies the viewing permission criterion, render the virtual note on the second client device.

2. The system of claim 1 , wherein
the client interface component is further configured to receive, from the second client device, another virtual note to be associated with the industrial asset, and
the rendering component is further configured to, in response to determining that the industrial asset is being presented on the first client device, render the other virtual note on the first client device.

3. The system of claim 1 , wherein the viewing permission criterion is at least one of an indication of a user role permitted to view the virtual note or an identity of a specified person permitted to view the virtual note.

4. The system of claim 1 , wherein at least one of the first client device or the second client device is a wearable computer or a handheld device.

5. The system of claim 1 , wherein the client interface component is configured to receive the indication of the industrial asset to which the virtual note pertains as at least one of a gesture detected by the first client device or a verbal command submitted to the first client device.

6. The system of claim 1 , wherein the rendering component is configured to render the virtual note on the second client device as an augmented reality overlay on or near the industrial asset.

7. The system of claim 1 , wherein the rendering component is configured to render a virtual note icon on the second client device as an augmented reality overlay on or near the industrial asset, and to render the virtual note on the second client device in response to detecting a gesture or verbal command indicating selection of the virtual note icon.

8. The system of claim 1 , wherein the client interface component is configured to receive the virtual note as spoken text submitted to the first client device.

9. The system of claim 1 , wherein the virtual note is an alphanumeric message.

10. The system of claim 1 , wherein
the client interface component is further configured to receive, from the first client device or a second client device, a spoken request for a current status of a specified industrial asset of the industrial system, and
the rendering component is configured to, in response to receipt of the spoken request, retrieve industrial data pertaining to the current status of the specified industrial asset and render, as an augmented reality overlay on the first client device or the second client device, the industrial data.

11. A method, comprising:
generating, by a system comprising a processor, on a first client device, an augmented reality presentation of one or more industrial assets located within a field of view of the first client device;
identifying, by the system, occurrence of a maintenance issue based on a determination that industrial data from the one or more industrial assets satisfies a condition indicative of the maintenance issue;
in response to the identifying:
retrieving, by the system, workflow data defining a workflow for correcting the maintenance issue, and
rendering, by the system, the workflow data on the first client device as part of the augmented reality presentation;

receiving, by the system from the first client device, a virtual note, an indication of an industrial asset, of the one or more industrial assets, to which the virtual note pertains, and a viewing permission criterion for the virtual note; and
in response to determining that the industrial asset is being rendered on a second client device, and that the second client device satisfies the viewing permission criterion, rendering, by the system, the virtual note on the second client device.

12. The method of claim 11 , wherein the receiving the viewing permission criterion comprises receiving at least one of an indication of a user role permitted to view the virtual note or an identity of a specified person permitted to view the virtual note.

13. The method of claim 11 , wherein at least one of the first client device or the second client device is a wearable computer or a handheld device.

14. The method of claim 11 , wherein the receiving the indication of the industrial asset to which the virtual note pertains comprises receiving the indication as at least one of a gesture detected by the first client device or a verbal command submitted to the first client device.

15. The method of claim 11 , wherein the rendering the virtual note on the second client device comprises rendering the virtual note as an augmented reality overlay positioned on or near the industrial asset within a field of view of the second client device.

16. The method of claim 11 , further comprising rendering, by the system, a virtual note icon as an augmented reality overlay on or near the industrial asset within a field of view of the second client device,
wherein the rendering the virtual note on the second client device comprises rendering the virtual note in response to detecting a gesture or verbal command indicating selection of the virtual note icon.

17. The method of claim 11 , wherein the receiving the virtual note comprises receiving the virtual note as spoken text submitted to the first client device.

18. The method of claim 10 , further comprising:
receiving, by the system from the first client device or the second client device, a spoken request for a current status of a specified industrial asset of the industrial system, and
in response to receipt of the spoken request:
retrieving, by the system, industrial data pertaining to the current status of the specified industrial asset, and
rendering, by the system as an augmented reality overlay on the first client device or the second client device, the industrial data.

19. A non-transitory computer-readable medium having stored thereon instructions that, in response to execution, cause a system comprising a processor to perform operations, the operations comprising:
rendering, on a first client device, an augmented reality presentation of one or more industrial assets located within a field of view of the first client device;
identifying an occurrence of a maintenance issue based on a determination that industrial data from the one or more

Assignees

• Rockwell Automation Tech Inc

Inventors

• Schmirler Paul D
• Nguyen Thong T
• Nicoll Alex L
• Vasko David

Classifications

• H04N23/90

  Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums · CPC title

• H04N23/698

  for achieving an enlarged field of view, e.g. panoramic image capture · CPC title

• H04N13/398

  Synchronisation thereof; Control thereof · CPC title

• H04N13/366

  using viewer tracking · CPC title

• H04N13/332

  Displays for viewing with the aid of special glasses or head-mounted displays [HMD] · CPC title