Multimodal search input for an industrial search platform

Patent metadata

Field	Value
Publication number	US-10318570-B2
Application number	US-201615240161-A
Country	US
Kind code	B2
Filing date	Aug 18, 2016
Priority date	Aug 18, 2016
Publication date	Jun 11, 2019
Grant date	Jun 11, 2019

Abstract

An industrial search and diagnosis system discovers available data items across heterogeneous data platforms that make up an industrial enterprise and indexes the data items in a unified searchable namespace. The system also indexes non-textual or multimedia input, including image data, video data, and audio data. This multimedia data can be indexed by the system in association with the relevant industrial systems (e.g., machines, automation systems, devices, production lines, etc.) so that the system can provide the multimedia data to the user when the relevant industrial systems are searched or invoked. The system also accepts and processes multimedia data from as search criteria or for diagnostic analysis. In an example analysis, users can submit images, audio, or video of an industrial machine or automation system, and the diagnosis system can identify an operational abnormality based on comparative analysis performed on the multimedia information.

First claim

What is claimed is:

1. A system for facilitating diagnosis of industrial performance issues, comprising:
a memory that stores executable components;
a processor, operatively coupled to the memory, that executes the executable components, the executable components comprising:
a transform component configured to
receive multimedia data captured by a client device,
correlate location information representing a location of the client device at a time that the multimedia data was received with respective locations of automation systems recorded in a federated data model to yield a correlation result,
determine an automation system, of the automation systems, represented by the multimedia data based on the correlation result, and
tag the multimedia data with contextual information identifying the automation system represented by the multimedia data;

an indexing component configured to index the multimedia data in a federated data model of an industrial enterprise in association with the automation system based on the contextual information;
an analysis component configured to identify a performance issue associated with the automation system based on an analysis of the multimedia data; and
a notification component configured to send notification data to one or more recipient client devices in response to identification of the performance issue, the notification data including at least one of a guidance image or a guidance video.

2. The system of claim 1 , wherein
the multimedia data is at least one of an image or a video, and
the transform component is configured to generate at least a portion of the contextual information based on an analysis of one or more identifying characteristics contained in the image or the video.

3. The system of claim 2 , wherein the one or more identifying characteristics comprise at least one of nameplate information extracted from the image or video or a visual property of the automation system identified in the image or video.

4. The system of claim 1 , wherein
the multimedia data comprises an audio maintenance recording recorded on the client device and submitted to the transform component, and
the notification component is further configured to deliver the maintenance recording to another client device in response to a determination that the other client device has moved within a defined distance from the automation system.

5. The system of claim 1 , wherein
the indexing component is configured to classify the multimedia data as reference data in a selected one of multiple categories defined in the federated data model, and
the multiple categories comprise at least a first category representing normal operation of the automation system and one or more second categories representing respective one or more abnormal conditions of the automation system.

6. The system of claim 5 , wherein
the multimedia data is first multimedia data,
the transform component is further configured to receive second multimedia data representing the automation system from the client device or another client device, and
the analysis component is configured to perform comparative analysis on the second multimedia data relative to the reference data, and to identify the performance issue associated with the automation system based on a result of the comparative analysis.

7. The system of claim 6 , wherein the comparative analysis comprises
inferring, based on analysis of the first multimedia data and the second multimedia data, at least one of vibration rates of a machine component or product throughput rates of the automation system for each of the first multimedia data and the second multimedia data, and
generating the result based on a comparison of the vibration rates or the product throughput rates for the first multimedia data and the second multimedia data.

8. The system of claim 6 , wherein the notification component is configured to select the one or more recipient client devices based on one or more notification rules that define one or more notification recipients that are associated with the automation system.

9. The system of claim 1 , wherein
the multimedia data comprises an audio maintenance recording recorded on the client device and submitted to the transform component,
the indexing component is configured to associate the maintenance recording with a human-machine interface terminal associated with the automation system identified by the transform component,
the executable components further comprise a device interface component configured to render an interactive link on the human-machine interface terminal indicating availability of the maintenance recording, and
the notification component is further configured to, in response to selection of the interactive link via the human-machine interface terminal, deliver the maintenance recording to the human-machine interface terminal.

10. The system of claim 5 , wherein the one or more abnormal conditions comprise at least one of a leaking pump, a slipping belt, or excessive machine vibration.

11. The system of claim 6 , wherein
the reference data has associated therewith metadata identifying a portion of an image or video represented by the second multimedia data that is to be analyzed for presence of a visual characteristic indicative of the performance issue, and
the analysis component is configured to perform the comparative analysis on the portion of the image or video indicated by the metadata.

12. A method for identifying industrial performance issues based on multimedia information, comprising:
receiving, by a system comprising at least one processor, multimedia data recorded by a client device;
receiving, by the system, location information identifying a location of the client device at a time that the multimedia data was received;
correlating, by the system, the location information with respective locations of automation systems recorded in a federated data model;
determining, by the system, an automation system, of the automation systems, represented by the multimedia data based on a result of the correlating;
tagging, by the system, the multimedia data with contextual information identifying the automation system represented by the multimedia data;
indexing, by the system and based on the contextual information, the multimedia data in a federated data model of an industrial enterprise in association with the automation system;
identifying, by the system based on an analysis performed on the multimedia data, a performance issue associated with the automation system; and
in response to the identifying, sending, by the system, notification data to one or more recipient client devices, the notification data including at least one of a guidance image or a guidance video.

13. The method of claim 12 , wherein
the receiving comprises receiving at least one of an image or a video of the automation system, and
the tagging comprises generating at least a portion of the contextual information based on an analysis of one or more identifying characteristics contained in the image or the video.

14. The method of claim 13 , wherein the tagging further comprises identifying, as the one or more identifying characteristics, at least one of nameplate information contained within the image or video or a visual property of the automation system recorded in the image or video.

15. The method of claim 12 , wherein the receiving comprises receiving, as the multimedia data, an audio maintenance recording recorded on the client device, and the method further comprises:
tracking, by the system, a location of another

Assignees

• Rockwell Automation Tech Inc

Inventors

• Billi-Duran Sharon M
• Como Christopher W
• Gray Edward A
• Reissner Kyle K
• Walter Jonathan D
• Singhai Mohit
• Reichard Douglas J
• Sandler Scott N
• Bliss Ronald E
• Pantaleano Michael J
• Cahalane Ryan
• Korpela Jessica L
• Mccleave Jr Bruce T

Classifications

• G06F16/41 Primary

  Indexing; Data structures therefor; Storage structures · CPC title

• G06F16/487

  using geographical or spatial information, e.g. location · CPC title