Interactive troubleshooting assistant

Patent metadata

Field	Value
Publication number	US-10901834-B2
Application number	US-201916352441-A
Country	US
Kind code	B2
Filing date	Mar 13, 2019
Priority date	Mar 13, 2019
Publication date	Jan 26, 2021
Grant date	Jan 26, 2021

Abstract

An interactive troubleshooting assistant and method for troubleshooting a system in real time to repair (fix) one or more problems in a system is disclosed. The interactive troubleshooting assistant and method may include receiving multimodal inputs from sensors, wearable devices, a person, etc. that may be input into a feature extractor including attention layers and pre-processing units of a cloud computing system hosted by one or more servers, such as a private cloud system. A pre-processing unit converts the raw multimodal input into a structed form so that an attention layer can give weights to features provided by the pre-processing unit according to their importance. The weighted extracted features may be provided to an actions predictor. The actions predictor generates the most suitable action based on the weighted extracted features generated by the feature extractor based on the multimodal inputs. After the most suitable action is performed, the interactive troubleshooting assistant considers new information from multimodal inputs so that the interactive troubleshooting assistant can provide the next recommended action. The interactive troubleshooting assistant may repeat these operations until the repair is completed.

First claim

We claim:

1. A method of troubleshooting a system for detecting a system failure and for dynamically recommending one or more actions for repairing the detected failure of the system, the method comprising:
receiving initial sensor inputs from internet of things (IOT) sensors and storing the initial sensor inputs as initial sensor data in a cloud server;
analyzing the initial sensor data to detect the system failure;
designating a worker to repair the detected system failure based on the analysis of the initial sensor data;
providing details of the detected system failure to the designated worker via a mobile computing device;
repeatedly performing operations until determining that the failure is repaired, the operations comprising:
receiving images from an imaging device and text from the mobile computing device and storing the images as image data and the text as text data in the cloud server, wherein the images show one or more components of the system and the text includes input from the designated worker;
receiving additional sensor inputs from internet of things (IOT) sensors and storing the additional sensor inputs as sensor data in the cloud server;
determining whether the failure is repaired based on one or more of the image data, text data, and sensor data;
upon determining that the failure is not repaired, extracting features from image data, text data, and sensor data, weighting the extracted features, and storing the weighted extracted features in the cloud server;
determining, in real time, an action based on the weighted extracted features; and
recommending the action to the designated worker via communication to the mobile computing device from the cloud server; and

communicating, from the mobile computing device of the designated worker to the cloud server, a message which confirms that the system is repaired.

2. The method of claim 1 , wherein the designated worker is a robot.

3. The method of claim 1 , wherein the mobile computing device comprises one of a smartphone, a tablet, a smartwatch, a smart glass, a smart helmet, and a laptop.

4. The method of claim 1 , wherein the extraction of features from image data comprises reducing the dimension of image data to extract features from the image data.

5. The method of claim 1 , wherein the extraction of features from text data comprises reducing the dimension of text data to extract features from the text data.

6. The method of claim 1 , wherein the extraction of features from sensor data comprises:
converting, for each sensor, sensor data into statistical measures including an arithmetic mean, median, standard deviation, standardized moment, kurtosis, skewness, entropy, and root mean square;
mapping, for each sensor, the statistical measures to a value that is a combination of these statistical measures; and
mapping each value from each sensor to another value which is a combination of the value from each sensor.

7. A non-transitory computer readable medium storing software comprising instructions executable by one or more computers, which, upon such execution, cause the one or more computers to:
receive initial sensor inputs from internet of things (IOT) sensors and storing the initial sensor inputs as initial sensor data in a cloud server;
analyze the initial sensor data to detect the system failure;
designate a worker to repair the detected system failure based on the analysis of the initial sensor data;
provide details of the detected system failure to the designated worker via a mobile computing device;
repeatedly perform operations until determining that the failure is repaired, the operations comprising:
receiving images from an imaging device and text from the mobile computing device and storing the images as image data and the text as text data in the cloud server, wherein the images show one or more components of the system and the text includes input from the designated worker;
receiving additional sensor inputs from internet of things (IOT) sensors and storing the additional sensor inputs as sensor data in the cloud server;
determining whether the failure is repaired based on one or more of the image data, text data, and sensor data;
upon determining that the failure is not repaired, extracting features from image data, text data, and sensor data, weighting the extracted features, and storing the weighted extracted features in the cloud server;
determining, in real time, an action based on the weighted extracted features; and
recommending the action to the designated worker via communication to the mobile computing device from the cloud server; and

communicate, from the mobile computing device of the designated worker to the cloud server, a message which confirms that the system is repaired.

8. The non-transitory computer-readable medium storing software of claim 7 , wherein the designated worker is a robot.

9. The non-transitory computer readable medium of claim 7 , wherein the mobile computing device comprises one of a smartphone, a tablet, a smartwatch, a smart glass, a smart helmet, and a laptop.

10. The non-transitory computer-readable medium storing software of claim 7 , wherein the extraction of features from image data comprises reducing the dimension of image data to extract features from the image data.

11. The non-transitory computer-readable medium storing software of claim 7 , wherein the extraction of features from text data comprises reducing the dimension of text data to extract features from the text data.

12. The non-transitory computer-readable medium storing software of claim 7 , wherein the extraction of features from sensor data comprises:
converting, for each sensor, sensor data into statistical measures including an arithmetic mean, median, standard deviation, standardized moment, kurtosis, skewness, entropy, and root mean square;
mapping, for each sensor, the statistical measures to a value that is a combination of these statistical measures; and
mapping each value from each sensor to another value which is a combination of the value from each sensor.

13. An interactive troubleshooting assistant for troubleshooting a system to detect a system failure and for dynamically recommending one or more actions for repairing the detected failure of the system, the interactive troubleshooting assistant comprising:
one or more computers and one or more storage devices storing instructions that are operable, when executed by one or more computers, to cause one or more computers to:
receive initial sensor inputs from internet of things (IOT) sensors and storing the initial sensor inputs as initial sensor data in a cloud server;
analyze the initial sensor data to detect the system failure;
designate a worker to repair the detected system failure based on the analysis of the initial sensor data;
provide details of the detected system failure to the designated worker via a mobile computing device;
repeatedly perform operations until determining that the failure is repaired, the operations comprising:
receiving images from an imaging device and text from the mobile computing device and storing the images as image data and the text as text data in the cloud server, wherein the images show one or more components of the system and the text includes input from the designated worker;
receiving additional sensor inputs from internet of things (IOT) sensors and storing the additional sensor inputs as sensor data in the cloud server;
determining whether the failure is repaired based on one or more of the image data, text data, and sensor data;
upon determining that the failure is not repaired, extracting features from image data,

Assignees

• Accenture Global Solutions Ltd

Inventors

• Abhinav Kumar
• Dubey Alpana
• Jain Sakshi
• Duncan Michael L
• Bhardwaj Nitish
• Singal Niyati

Classifications

• G06F40/20 Primary

  Natural language analysis (semantic analysis of natural language G06F40/30) · CPC title

• G06V10/82

  using neural networks · CPC title

• G06V10/764

  using classification, e.g. of video objects · CPC title

• G06F11/0793 Primary

  Remedial or corrective actions (recovery from an exception in an instruction pipeline G06F9/3861; by retry G06F11/1402; for recovering from a failure of a protocol instance or entity H04L69/40) · CPC title

• G06V20/20

  in augmented reality scenes · CPC title