Imitation learning system

Patent metadata

Field	Value
Publication number	US-2021081752-A1
Application number	US-202016931211-A
Country	US
Kind code	A1
Filing date	Jul 16, 2020
Priority date	Sep 13, 2019
Publication date	Mar 18, 2021
Grant date	—

Abstract

Apparatuses, systems, and techniques to identify a goal of a demonstration. In at least one embodiment, video data of a demonstration is analyzed to identify a goal. Object trajectories identified in the video data are analyzed with respect to a task predicate satisfied by a respective object trajectory, and with respect to motion predicate. Analysis of the trajectory with respect to the motion predicate is used to assess intentionality of a trajectory with respect to the goal.

First claim

What is claimed is:

1 . A computer-implemented method, comprising:
segmenting video data into at least a first segment and a second segment; the first segment comprising video data representative of a first trajectory of a first object manipulated in a demonstration, the second segment comprising video data representative of a second trajectory of a second object manipulated in the demonstration; identifying a motion predicate satisfied by the first trajectory, wherein the motion predicate is identified based, at least in part, on a determination that movement of the first object on the first trajectory enabled movement of the second object on the second trajectory; identifying a task predicate satisfied by the second trajectory, based at least in part on the second trajectory satisfying a logical condition defined in a domain definition; and identifying a goal of the demonstration based at least in part on the task predicate.

2 . The computer-implemented method of claim 1 , further comprising:
using the task predicate to identify the goal based, at least in part, on determining that the second trajectory does not enable another trajectory of an object manipulated in the demonstration.

3 . The computer-implemented method of claim 1 , further comprising:
causing one or more robotic manipulation devices to achieve the goal by performing one or more manipulations to satisfy one or more predicates of the goal.

4 . The computer-implemented method of claim 3 , wherein the goal is achieved by the one or more robotic manipulation devices in an environment other than an environment in which the demonstration was performed.

5 . The computer-implemented method of claim 3 , wherein at least one manipulation of an object in the demonstration is excluded from the one or more manipulations by the robotic manipulation devices.

6 . The computer-implemented method of claim 1 , wherein the goal is identified from a set of task predicates that excludes an additional task predicate satisfied by the first trajectory.

7 . The computer-implemented method of claim 6 , wherein the additional task predicate is excluded based, at least in part, on the determination that movement of the first object on the first trajectory enabled movement of the second object on the second trajectory.

8 . The computer-implemented method of claim 1 , wherein the domain definition comprises a list of objects and a list of task predicates satisfiable by manipulation of objects in the list of objects.

9 . A system, comprising:
at least one processor; and a memory comprising instructions that, as a result of execution by the at least one processor, cause the system to at least:
segment video data into a first segment and a second segment, the first segment comprising video data representative of a first trajectory of a first object manipulated in a demonstration, the second segment comprising video data representative of a second trajectory of a second object manipulated in the demonstration;
identify a motion predicate satisfied by manipulation of the first object in accordance with the first trajectory, wherein the motion predicate is identified based, at least in part, on a determination that manipulation of the first object in accordance with the first trajectory enabled motion of the second object in accordance with the second trajectory;
identify a task predicate satisfied by manipulation of the second object in accordance with the second trajectory; and
identify a goal of the demonstration based at least in part on the task predicate.

10 . The system of claim 9 , wherein the task predicate is identified based at least in part on determining that the second trajectory does not enable another trajectory.

11 . The system of claim 9 , the memory comprising further instructions that, as a result of execution by the at least one processor, cause the system to at least cause one or more robotic manipulation devices to achieve the goal.

12 . The system of claim 11 , wherein the goal is achieved by the robotic manipulation devices in an environment other than an environment in which the demonstration was performed.

13 . The system of claim 9 , wherein the goal of the demonstration is identified based, at least in part, on an estimate of intentionality of the task predicate.

14 . The system of claim 13 , wherein the estimate of intentionality is based, at least in part, on a relationship between first trajectory and the second trajectory.

15 . The system of claim 9 , wherein a first task predicate satisfied by the first trajectory is assessed low intentionality based, at least in part, on the second trajectory being enabled by the first trajectory.

16 . A non-transitory computer-readable medium comprising instructions that, as a result of being executed by one or more processors of a computer system, cause the computer system to:
divide video data into a first segment and a second segment, the first segment comprising video data representative of a first trajectory of a first object manipulated in a demonstration, the second segment comprising video data representative of a second trajectory of a second object; identify a motion predicate satisfied by the first trajectory, wherein the motion predicate is identified based, at least in part, on a determination that manipulation of the first object enabled the second trajectory; identify a task predicate satisfied by the second trajectory; and identify a goal of the demonstration based at least in part on the task predicate.

17 . The non-transitory computer-readable medium of claim 16 , wherein the task predicate is used to identify the goal based, at least in part, on determining that the second trajectory does not enable another trajectory.

18 . The non-transitory computer-readable medium of claim 16 , comprising further instructions that, as a result of execution by the one or more processors of the computer system, cause the computer system to achieve the goal by instructing a robotic manipulation device to manipulate objects to achieve task predicates of the goal.

19 . The non-transitory computer-readable medium of claim 16 , wherein the goal of the demonstration is identified based, at least in part, on an estimated intentionality of the task predicate.

20 . The non-transitory computer-readable medium of claim 19 , wherein the estimated intentionality is based, at least in part, on analysis of the first trajectory with respect to the second trajectory.

21 . A robotic device, comprising:
one or more robotic manipulation devices; at least one processor; and a memory comprising instructions that, as a result of execution by the at least one processor, cause the robotic device to at least: obtain instructions to perform one or more manipulations of the one or more robotic arms to achieve a goal, wherein the goal determined by:
segmenting video data of a demonstration into a first segment and a second segment, the first segment comprising video data representative of a first trajectory of a first object, the second segment comprising video data representative of a second trajectory of a second object;
identifying a motion predicate satisfied by the first trajectory, wherein the motion predicate is identified based, at least in part, on a determination that the first trajectory enabled the second trajectory;
identifying a task predicate satisfied by manipulation of the second object in accordance with the second trajectory; and
identifyin

Assignees

• Nvidia Corp

Inventors

• Chao Yu-Wei
• Huang De-An
• Paxton Christopher Jason
• Garg Animesh
• Fox Dieter

Classifications

• G06N3/045

  Combinations of networks · CPC title

• G06N3/0985

  Hyperparameter optimisation; Meta-learning; Learning-to-learn · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06N3/08

  Learning methods · CPC title