Learning robotic tasks using one or more neural networks

Patent metadata

Field	Value
Publication number	US-12430702-B2
Application number	US-202117458221-A
Country	US
Kind code	B2
Filing date	Aug 26, 2021
Priority date	Jan 23, 2018
Publication date	Sep 30, 2025
Grant date	Sep 30, 2025

Abstract

Various embodiments enable a robot, or other autonomous or semi-autonomous device or system, to receive data involving the performance of a task in the physical world. The data can be provided as input to a perception network to infer a set of percepts about the task, which can correspond to relationships between objects observed during the performance. The percepts can be provided as input to a plan generation network, which can infer a set of actions as part of a plan. Each action can correspond to one of the observed relationships. The plan can be reviewed and any corrections made, either manually or through another demonstration of the task. Once the plan is verified as correct, the plan (and any related data) can be provided as input to an execution network that can infer instructions to cause the robot, and/or another robot, to perform the task.

First claim

What is claimed is:

1. A processor comprising:
processing circuitry to use a first portion of one or more neural networks to detect one or more objects in one or more images and a second portion of the one or more neural networks to identify one or more spatial relationships between the one or more objects; and
cause one or more tasks to be performed based, at least in part, on the identified one or more spatial relationships.

2. The processor of claim 1 , wherein the processing circuitry is further to cause one or more tasks to be performed based, at least in part, on the identified one or more spatial relationships, in substantially a same way in which the one or more tasks are observed being performed.

3. The processor of claim 1 , wherein the processing circuitry is further to cause one or more tasks to be performed based, at least in part, on the identified one or more spatial relationships, in substantially a different way than the one or more tasks are observed being performed.

4. The processor of claim 1 , wherein one or more tasks are performed based, at least in part, on the identified one or more spatial relationships, at least in part, by a robot, a semi-automated machine, or a human.

5. The processor of claim 1 , wherein the processing circuitry is further to generate a plan including one or more steps to perform one or more tasks based, at least in part, on the identified one or more spatial relationships.

6. The processor of claim 5 , wherein the one or more steps of the plan are human-readable.

7. The processor of claim 1 , wherein the one or more neural networks are to identify a task being performed based, at least in part, on the identified spatial relationships.

8. A method, comprising:
using a first portion of one or more neural networks to detect one or more objects in one or more images and a second portion of the one or more neural networks to identify one or more spatial relationships between the one or more objects; and
cause one or more tasks to be performed based, at least in part, on the identified one or more spatial relationships.

9. The method of claim 8 , further comprising:
causing one or more tasks to be performed based, at least in part, on the identified one or more spatial relationships, in substantially a same way in which the one or more tasks are observed being performed.

10. The method of claim 8 , further comprising:
causing one or more tasks to be performed based, at least in part, on the identified one or more spatial relationships, in substantially a different way than the one or more tasks are observed being performed.

11. The method of claim 8 , wherein one or more tasks are performed based, at least in part, on the identified one or more spatial relationships, at least in part, by a robot, a semi-automated machine, or a human.

12. The method of claim 8 , further comprising: generating a plan including one or more steps to perform one or more tasks based, at least in part, on the identified one or more spatial relationships.

13. The method of claim 12 , wherein the one or more steps of the plan are human-readable.

14. The method of claim 13 , further comprising:
providing the plan, including the human-readable steps, to one or more second neural networks to generate instructions for a robotic device to perform the one or more tasks.

15. The method of claim 8 , wherein the one or more neural networks are to identify a task being performed based, at least in part, on the identified one or more spatial relationships.

16. A system, comprising:
one or more cameras to capture image data of one or more motions of one or more objects; and
one or more processors to use a first portion of one or more neural networks to detect the one or more objects in the captured image data and a second portion of the one or more neural networks to identify one or more one or more spatial relationships between the one or more objects, and
cause one or more tasks to be performed based, at least in part, on the identified one or more spatial relationships.

17. The system of claim 16 , wherein the one or more processors are further to cause one or more tasks to be performed based, at least in part, on the one or more identified one or more spatial relationships, in substantially a same way in which the one or more tasks are observed being performed.

18. The system of claim 16 , wherein the one or more processors are further to cause one or more tasks to be performed based, at least in part, on the one or more identified one or more spatial relationships, in substantially a different way than the one or more tasks are observed being performed.

19. The system of claim 16 , wherein one or more tasks are performed based, at least in part, on the one or more identified one or more spatial relationships, at least in part, by a robot, a semi-automated machine, or a human.

20. The system of claim 16 , wherein the one or more processors are further to generate a plan including one or more steps to perform one or more tasks based, at least in part, on the one or more identified relationships, and wherein the one or more steps of the plan are human-readable.

Assignees

• Nvidia Corp

Inventors

• Tremblay Jonathan
• Birchfield Stan
• Tyree Stephen
• To Thang
• Kautz Jan
• Molchanov Artem

Classifications

• G06N3/084

  Backpropagation, e.g. using gradient descent · CPC title

• G05B2219/40494

  Neural network for object trajectory prediction, fuzzy for robot path · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G06N3/09

  Supervised learning · CPC title