Systems and method for human-assisted robotic industrial inspection

Patent metadata

Field	Value
Publication number	US-10775314-B2
Application number	US-201715809531-A
Country	US
Kind code	B2
Filing date	Nov 10, 2017
Priority date	Nov 10, 2017
Publication date	Sep 15, 2020
Grant date	Sep 15, 2020

Abstract

An asset inspection system includes a robot and a server. The robot collects inspection data corresponding to an asset. The server, includes a user interface, a processor, and a memory. The memory includes instructions that, when executed by the processor, cause the processor to receive the inspection data from the robot, display the inspection data via the user-interface, receive feedback on the inspection data via the user interface, generate a human-assisted inspection based on the received feedback, analyze the inspection data via a trained model, generate an automated inspection based on the analysis by the trained model, combine the automated inspection and the human-assisted inspection to generate an inspection report, and transmit the inspection report for review.

First claim

The invention claimed is:

1. An asset inspection system, comprising:
a robot configured to collect inspection data of an asset; and
a server, comprising:
a user interface;
a processor; and
a memory comprising instructions that, when executed by the processor, cause the processor to:
receive the inspection data from the robot;
display the inspection data via the user interface;
receive feedback on one or more aspects of the inspection data via the user interface;
generate, based on the feedback, human-assisted inspection results corresponding to a first set of measured quantitative parameters for the one or more aspects of the inspection data as determined based on the received feedback;
analyze the inspection data via a trained model;
generate, via analysis of the trained model, automated inspection results corresponding to a second set of measured quantitative parameters for the one or more aspects of the inspection data as determined based on the analysis by the trained model;
assign first weight values to the first set of measured quantitative parameters and second weight values to the second set of measured quantitative parameters, wherein the first set of measured quantitative parameters has an associated first confidence score and the second set of measured quantitative parameters has an associated second confidence score:
combine the first set of measured quantitative parameters weighted by the first weight values and the second set of measured quantitative parameters weighted by the second weight values to generate an inspection report having an overall confidence score and including the weighted, combined human-assisted inspection results and the automated inspection results; and
transmit the inspection report to a client.

2. The asset inspection system of claim 1 , wherein the server is disposed remote from the robot, and wherein the robot and the server are configured to communicate with one another via a cloud interface.

3. The asset inspection system of claim 1 , wherein the automated inspection results and the human-assisted inspection results are combined via a decision making operation.

4. The asset inspection system of claim 1 , wherein the robot is configured to process, partially process, or pre-process the inspection data.

5. The asset inspection system of claim 1 , wherein the robot is configured to transmit raw collected inspection data to the server.

6. The asset inspection system of claim 1 , wherein the server is configured to train the trained model using the feedback received via the user interface.

7. The asset inspection system of claim 1 , wherein the first weight values and the second weight values are adjusted over time based on a size of training data included in the trained model to facilitate adaptive incorporation of the human-assisted inspection results and the automated inspection results in the inspection report.

8. The asset inspection system of claim 1 , wherein the one or more aspects include a sufficiency indication of the inspection data.

9. The asset inspection system of claim 1 , wherein the one or more aspects include indicators of a condition of the asset.

10. An asset inspection system, comprising a server, the server comprising:
a user interface;
a processor; and
a memory comprising instructions that, when executed by the processor, cause the processor to:
receive inspection data from a robot;
display the inspection data via the user interface;
receive feedback on one or more aspects of the inspection data via the user interface;
generate, based on the feedback, human-assisted inspection results corresponding to a first set of measured quantitative parameters for the one or more aspects of the inspection data as determined based on the received feedback;
analyze the inspection data via a trained model;
generate, via analysis of the trained model, automated inspection results corresponding to a second set of measured quantitative parameters for the one or more aspects of the inspection data as determined based on the analysis by the trained model;
assign first weight values to the first set of measured quantitative parameters and second weight values to the second set of measured quantitative parameters, wherein the first set of measured quantitative parameters has an associated first confidence score and the second set of measured quantitative parameters has an associated second confidence score;
combine the first set of measured quantitative parameters weighted by the first weight values and the second set of measured quantitative parameters weighted by the second weight values to generate an inspection report having an overall confidence score and including the weighted, combined human-assisted inspection results and the automated inspection results; and
transmit the inspection report to a client.

11. The asset inspection robot of claim 10 , wherein the automated inspection results and the human-assisted inspection results are combined via a decision making operation.

12. The asset inspection robot of claim 10 , wherein the feedback comprises identification of one or more features in an asset.

13. The asset inspection robot of claim 10 , wherein the feedback comprises a determination as to a sufficiency of the inspection data.

14. The asset inspection robot of claim 10 , wherein the server is configured to train the trained model using the feedback received via the user interface.

15. A method of inspecting an asset, comprising:
collecting, via a robot, inspection data related to the asset;
transmitting the inspection data to a server;
displaying the inspection data via a user interface;
receiving feedback on one or more aspects of the inspection data via the user interface;
generating, based on the inspection data, human-assisted inspection results corresponding to a first set of measured quantitative parameters for the one or more aspects of the inspection data as determined based on the received feedback;
analyzing the inspection data via a trained model;
generating, via analysis of the trained model, automated inspection results corresponding to a second set of measured quantitative parameters for the one or more aspects of the inspection data as determined based on the analysis by the trained model;
assign first weight values to the first set of measured quantitative parameters and second weight values to the second set of measured quantitative parameters, wherein the first set of measured quantitative parameters has an associated first confidence score and the second set of measured quantitative parameters has an associated second confidence score;
combining the first set of measured quantitative parameters weighted by the first weight values and the second set of measured quantitative parameters weighed by the second weight values to generate an inspection report having an overall confidence score and including the weighted, combined human-assisted inspection results and the automated inspection results; and
transmitting the inspection report to a client.

16. The method of claim 15 , wherein the automated inspection results and the human-assisted inspection results are combined via a decision making operation.

17. The method of claim 15 , comprising retraining the trained model using the feedback received via the user interface.

Assignees

• Gen Electric

Inventors

• Tan Huan
• Zhang Li
• Patrick Romano
• Jain Arpit
• Zhao Guangliang
• Tu Jilin

Classifications

• G01N21/9515

  Objects of complex shape, e.g. examined with use of a surface follower device (measuring contours and curvatures G01B11/24) · CPC title

• Y04S40/18

  Network protocols supporting networked applications, e.g. including control of end-device applications over a network · CPC title

• G06F3/03547

  Touch pads, in which fingers can move on a surface · CPC title

• G01N21/8851 Primary

  Scan or image signal processing specially adapted therefor, e.g. for scan signal adjustment, for detecting different kinds of defects, for compensating for structures, markings, edges (G01N21/8806 and G01N21/93 - G01N21/95692 take precedence; optical measurement of dimensions G01B11/00; optical scanning G02B26/10; image transformation G06T3/00; computerised image enhancement G06T5/00; image processing per se for flaw detection G06T7/0002) · CPC title

• G01N21/8803 Primary

  Visual inspection (measuring projectors G01B9/08) · CPC title