Systems and method for robotic learning of industrial tasks based on human demonstration

Patent metadata

Field	Value
Publication number	US-10913154-B2
Application number	US-201815860377-A
Country	US
Kind code	B2
Filing date	Jan 2, 2018
Priority date	Jan 2, 2018
Publication date	Feb 9, 2021
Grant date	Feb 9, 2021

Abstract

A system for performing industrial tasks includes a robot and a computing device. The robot includes one or more sensors that collect data corresponding to the robot and an environment surrounding the robot. The computing device includes a user interface, a processor, and a memory. The memory includes instructions that, when executed by the processor, cause the processor to receive the collected data from the robot, generate a virtual recreation of the robot and the environment surrounding the robot, receive inputs from a human operator controlling the robot to demonstrate an industrial task. The system is configured to learn how to perform the industrial task based on the human operator's demonstration of the task, and perform, via the robot, the industrial task autonomously or semi-autonomously.

First claim

The invention claimed is:

1. A system for performing industrial tasks, comprising:
a robot, comprising one or more sensors configured to collect data corresponding to the robot and an environment surrounding the robot; and
a computing device disposed remote from the robot and configured to communicate with the robot via a cloud interface, the computing device comprising:
a user interface;
a processor; and
a memory comprising instructions that, when executed by the processor, cause the processor to:
receive, via the cloud interface, the collected data from the robot;
generate a virtual recreation of the robot and the environment surrounding the robot;
receive, via the user interface, inputs from a human operator controlling the robot to demonstrate an industrial task;

wherein the system is configured to:
learn how to perform the industrial task based on the human operator's demonstration of the task; and
perform, via the robot, the industrial task autonomously or semi-autonomously.

2. The system for performing industrial tasks of claim 1 , wherein the user interface comprises a virtual reality headset and a controller.

3. The system for performing industrial tasks of claim 1 , wherein the user interface comprises a desktop computer, a laptop computer, a tablet, a computer monitor, a television, a projector, or some combination thereof.

4. The system for performing industrial tasks of claim 1 , wherein the system is configured to:
generate a motion plan before beginning to perform the industrial task;
present the motion plan to the human operator via the operator interface; and
receive feedback on the motion plan via the operator interface.

5. The system for performing industrial tasks of claim 1 , comprising a remote server, wherein the remote server is configured to:
store data corresponding to the robot and the environment surrounding the robot;
learn how to perform the industrial task based on the human operator's demonstration of the task; and
autonomously or semi-autonomously control the robot during performance of the industrial task.

6. The system for performing industrial tasks of claim 1 , wherein the robot comprises a task planning component configured to plan the robot's performance of the industrial task.

7. A processor-implemented learning system, comprising:
a processor; and
a memory comprising instructions that, when executed by the processor, cause the processor to:
receive, via a cloud interface, inputs from a human operator controlling a robot as a demonstration of an industrial task, wherein the human operator is disposed remote from the robot;
learn how to control the robot to perform the industrial task based on the demonstration of the industrial task by the human operator; and
control the robot to perform the industrial task autonomously or semi-autonomously.

8. The processor-implemented learning system of claim 7 , wherein the instructions cause the processor to:
receive feedback from the human operator during the performance of the task or following performance of the task; and
relearn how to control the robot to perform the industrial task based on the feedback provided by the human operator.

9. The processor-implemented learning system of claim 7 , wherein the processor-implemented learning system is disposed within a server remote from the robot, and in communication with the robot.

10. The processor-implemented learning system of claim 7 , wherein the processor-implemented learning system is distributed over a plurality of servers in communication with the robot.

11. A method of performing an industrial task, comprising:
receiving inputs from a human operator demonstrating performance of an industrial task by a robot, wherein the robot is disposed remote from the human operator;
learning how to perform the industrial task based on the demonstration of the industrial task;
generating a motion plan for performing the industrial task;
displaying the motion plan to the human operator for approval; and
performing, via the robot, the industrial task upon approval of the motion plan.

12. The method of claim 11 , comprising:
collecting data corresponding to the robot and an environment surrounding the robot;
generating a virtual recreation of the robot and the environment surrounding the robot;
displaying the virtual recreation of the robot and the environment surrounding the robot to the human operator.

13. The method of claim 11 , wherein learning how to perform the industrial task based on the demonstration of the industrial task comprises generating a definition of the industrial task based on the human operator's demonstration of the industrial task, wherein the definition of the industrial task comprises a starting state, a goal state, one or more intermediate states, and one or more task constraints.

14. The method of claim 11 , comprising generalizing the demonstration of the industrial task by the human operator.

15. The method of claim 11 , comprising:
receiving feedback from the human operator; and
relearning the industrial task based on the feedback.

16. The system for performing industrial tasks of claim 1 , wherein learning how to perform the industrial task based on the human operator's demonstration of the task comprises generating a definition of the industrial task based on the human operator's demonstration of the industrial task.

17. The system for performing industrial tasks of claim 16 , wherein the definition of the industrial task comprises a starting state, a goal state, one or more intermediate states, and one or more task constraints.

18. The system for performing industrial tasks of claim 1 , wherein the instructions, when executed by the processor, cause the processor to:
present, via the operator interface, the autonomous or semi-autonomous performance of the industrial task to the human operator for the human operator to supervise; and
receive, via the operator interface, feedback from the human operator on the autonomous or semi-autonomous performance of the industrial task.

19. The system for performing industrial tasks of claim 18 , wherein the system is configured to relearn how to perform the industrial task based on the feedback from the human operator.

20. The processor-implemented learning system of claim 7 , wherein learning how to control the robot to perform the industrial task based on the demonstration of the industrial task by the human operator comprises identifying a starting state, a goal state, one or more intermediate states, and one or more task constraints based on the demonstration of the industrial task.

Assignees

• Gen Electric

Inventors

• Tan Huan
• Hoare John Robert
• Foehner Justin Michael
• Gray steven robert
• Sen Shiraj
• Patrick Romano

Classifications

• B25J9/1671 Primary

  characterised by simulation, either to verify existing program or to create and verify new program, CAD/CAM oriented, graphic oriented programming systems · CPC title

• G05B2219/40323

  Modeling robot environment for sensor based robot system · CPC title

• G05B2219/40099

  Graphical user interface for robotics, visual robot user interface · CPC title

• G05B2219/40515

  Integration of simulation and planning · CPC title

• G05B2219/40131

  Virtual reality control, programming of manipulator · CPC title