Learning robotic tasks using one or more neural networks

Patent metadata

Field	Value
Publication number	US-11941719-B2
Application number	US-201916255038-A
Country	US
Kind code	B2
Filing date	Jan 23, 2019
Priority date	Jan 23, 2018
Publication date	Mar 26, 2024
Grant date	Mar 26, 2024

Abstract

Various embodiments enable a robot, or other autonomous or semi-autonomous device or system, to receive data involving the performance of a task in the physical world. The data can be provided as input to a perception network to infer a set of percepts about the task, which can correspond to relationships between objects observed during the performance. The percepts can be provided as input to a plan generation network, which can infer a set of actions as part of a plan. Each action can correspond to one of the observed relationships. The plan can be reviewed and any corrections made, either manually or through another demonstration of the task. Once the plan is verified as correct, the plan (and any related data) can be provided as input to an execution network that can infer instructions to cause the robot, and/or another robot, to perform the task.

First claim

What is claimed is:

1. A computer-implemented method, comprising:
receiving image data representative of a task being physically performed;
inferring, using the image data as input to a perception neural network, a relationship between at least two objects resulting from performance of the task;
inferring, using the relationship as input to a plan generation neural network, a plan corresponding to the relationship between the at least two objects, the plan providing at least a human-readable representation of the task;
receiving confirmation of the plan; and
inferring, using an execution neural network and the plan, an instruction readable by a robotic device to cause the robotic device, upon execution of the instruction, to perform the task.

2. The computer-implemented method of claim 1 , further comprising:
inferring, using the image data as input to an object detection network, a set of belief maps representative of the at least two objects; and
identifying location probabilities for one or more features of the at least two objects from the belief maps.

3. The computer-implemented method of claim 2 , further comprising:
inferring, using the location probabilities as input to a relationship inference network, the relationship between the at least two objects.

4. The computer-implemented method of claim 1 , further comprising:
providing the instruction to a control system of the robotic device, the robotic device storing a set of pre-scripted behaviors enabling the robotic device to perform the task according to the instruction.

5. The computer-implemented method of claim 1 , further comprising:
causing the robotic device to perform the task using the instruction.

6. A computer-implemented method, comprising:
receiving data representative of a task to be performed by an automated device;
inferring, using a first neural network and the received data, a plan corresponding to the task, the plan providing at least a human-readable representation of the task; and
causing the task to be performed by the automated device using a second neural network and the plan corresponding to the task.

7. The computer-implemented method of claim 6 , further comprising:
capturing the data using at least one sensor of the automated device, the data including at least one of image data or video data representative of a physical demonstration of the task.

8. The computer-implemented method of claim 6 , further comprising:
inferring, using the data as input to a perception neural network, a relationship between at least two objects resulting from performance of the task.

9. The computer-implemented method of claim 8 , further comprising:
inferring, using the data as input to an object detection network, a set of belief maps indicative of locations of the at least two objects; and
identifying location probabilities for one or more features of the at least two objects from the belief maps.

10. The computer-implemented method of claim 9 , further comprising:
inferring, using the location probabilities as input to a relationship inference network, the relationship between the at least two objects.

11. The computer-implemented method of claim 8 , wherein the first neural network is a plan generation neural network, and further comprising:
inferring, using the relationship as input to the plan generation neural network, the plan corresponding to the task, the human-readable representation identifying at least one action corresponding to the relationship between the at least two objects.

12. The computer-implemented method of claim 11 , further comprising:
providing the human-readable representation for review by a human reviewer; and
causing the task to be performed by the automated device in response to receiving confirmation of the human-readable representation.

13. The computer-implemented method of claim 12 , wherein the human-readable representation is capable of being updated by capturing additional data for another physical demonstration of the task or through a manual updating by the human reviewer.

14. The computer-implemented method of claim 6 , wherein the second neural network is an execution neural network, and further comprising:
inferring, using the execution neural network, an instruction readable by the automated device to cause the automated device, upon execution of the instruction, to perform the task.

15. The computer-implemented method of claim 6 , wherein the data is captured using at least one of a digital camera, stereoscopic camera, infrared image sensor, structured light camera, depth sensor, ultrasonic sensor, LIDAR detector, microphone, motion capture system, or motion detector.

16. A system, comprising:
at least one processor; and
memory including instructions that, when executed by the at least one processor, cause the system to:
receive data representative of a task to be performed by an automated device;
infer, using a first neural network and the received data, a plan corresponding to the task, the plan providing at least a human-readable representation of the task; and
cause the task to be performed by the automated device using a second neural network and the plan corresponding to the task.

17. The system of claim 16 , wherein the instructions when executed further cause the system to:
capture the data using at least one sensor of the automated device, the data including at least one of image data or video data representative of a physical demonstration of the task.

18. The system of claim 16 , wherein the instructions when executed further cause the system to:
infer, using the data as input to a perception neural network, a relationship between at least two objects resulting from performance of the task.

19. The system of claim 18 , wherein the instructions when executed further cause the system to:
infer, using the data as input to an object detection network, a set of belief maps indicative of locations of the at least two objects;
identify location probabilities for one or more features of the at least two objects from the belief maps; and
infer, using the location probabilities as input to a relationship inference network, the relationship between the at least two objects.

20. The system of claim 16 , wherein the first neural network is a plan generation neural network, and wherein the instructions when executed further cause the system to:
infer, using the relationship as input to the plan generation neural network, the plan corresponding to the task, the human-readable representation identifying at least one action corresponding to the relationship between the at least at least two objects.

Assignees

• Nvidia Corp

Inventors

• Tremblay Jonathan
• Birchfield Stan
• Tyree Stephen
• To Thang
• Kautz Jan
• Molchanov Artem

Classifications

• G06N3/084

  Backpropagation, e.g. using gradient descent · CPC title

• G05B2219/40494

  Neural network for object trajectory prediction, fuzzy for robot path · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G06N3/09

  Supervised learning · CPC title