Data generation of robotic devices performing tasks

Patent metadata

Field	Value
Publication number	US-12539609-B2
Application number	US-202318239601-A
Country	US
Kind code	B2
Filing date	Aug 29, 2023
Priority date	Aug 29, 2023
Publication date	Feb 3, 2026
Grant date	Feb 3, 2026

Abstract

Apparatuses, systems, and techniques to generate data to train a robotic device to perform tasks. In at least one embodiment, one or more first videos of a robotic device performing a task is used to generate one or more second videos of the robotic device performing the task differently than depicted in the one or more first videos.

First claim

What is claimed is:

1 . One or more processors, comprising:
circuitry to: identify, from a first video of a robotic device performing a task in a first environment, one or more trajectories of at least one of one or more portions of the robotic device or one or more objects in the first environment; determine one or more locations for the at least one of the one or more portions of the robotic device or the one or more objects in a second environment; modify, based on the one or more locations, at least one trajectory of the one or more trajectories to generate one or more modified trajectories; and generate a second video of the robotic device performing the task depicted in the first video using the one or more modified trajectories.

2 . The one or more processors of claim 1 , wherein the circuitry is to:
segment the first video into one or more subtasks, wherein each subtask of the one or more subtasks comprises at least a trajectory of the robotic device; generate a new scene comprising an object different from an object used by the robotic device to perform the task in the first video; and generate the second video comprising one or more transformed trajectories such that the robotic device performs the task in the new scene differently than depicted in the first video.

3 . The one or more processors of claim 1 , wherein the circuitry is to:
use one or more cameras to record one or more demonstrations performed by the robotic device in a simulator, wherein the one or more demonstrations comprise a task that is performed differently than the task depicted in the first video.

4 . The one or more processors of claim 1 , wherein the circuitry is to train one or more neural networks using the first video and the second video to infer motion of the robotic device.

5 . The one or more processors of claim 1 , wherein the circuitry is to generate the second video without using teleoperation.

6 . The one or more processors of claim 1 , wherein the circuitry is to generate the second video after a predetermined threshold of successful simulations is met or exceeded.

7 . A method, comprising:
identifying, from a first video of a robotic device performing a task in a first environment, one or more trajectories of at least one of one or more portions of the robotic device or one or more objects in the first environment; determining one or more locations for the at least one of the one or more portions of the robotic device or the one or more objects in a second environment; modifying, based on the one or more locations, at least one trajectory of the one or more trajectories to generate one or more modified trajectories; and generating a second video of the robotic device performing the task depicted in the first video using the one or more modified trajectories.

8 . The method of claim 7 , further comprising generating the second video by:
modifying the one or more trajectories identified in the first video based, at least in part, on adjustments to a scene from the first video, wherein adjustments to the scene comprises: changing starting locations of the one or more objects, replacing the one or more objects with different objects, and/or changing the robotic device to a different robotic device.

9 . The method of claim 7 , further comprising:
using a simulator to perform the task differently than depicted in the first video to indicate whether the task performed differently is a successful demonstration.

10 . The method of claim 7 , wherein the first video comprise one or more demonstrations generated using teleoperation; and
wherein the second video comprises one or more demonstrations generated without using teleoperation.

11 . The method of claim 7 , further comprising using the first video and the second video to train a policy to control the robotic device to grasp the one or more objects.

12 . The method of claim 7 , further comprising interpolating a starting location of the one or more modified trajectories and a starting location of the robotic device.

13 . A system, comprising:
one or more processors to: identify, from a first video of a robotic device performing a task in a first environment, one or more trajectories of at least one of one or more portions of the robotic device or one or more objects in the first environment; determine one or more locations for the at least one of the one or more portions of the robotic device or the one or more objects in a second environment; modify, based on the one or more locations, at least one trajectory of the one or more trajectories to generate one or more modified trajectories; and generate a second video of the robotic device performing the task depicted in the first video using the one or more modified trajectories.

14 . The system of claim 13 , wherein the one or more processors are to:
use a subset of the first video to generate the second video based, at least in part, on one or more subtasks of the task performed by the robotic device.

15 . The system of claim 13 , wherein the second video comprise one or more recorded demonstrations generated by a simulator.

16 . The system of claim 13 , wherein the one or more processors are to:
segment the first video to a plurality of segments, wherein each segment includes an object and a trajectory of the robotic device to grasp the object; and generate a new trajectory of the robotic device to grasp a different object based, at least in part, on the trajectory of the robotic device to grasp the object from the first video.

17 . The system of claim 13 , wherein the second video comprises one or more contexts that are different from one or more contexts of the first video.

18 . The system of claim 13 , wherein the robotic device in the second video comprises a different arm connected to the robotic device than one or more arms of the robotic device in the first video.

19 . The system of claim 13 , wherein the one or more processors are to generate the second video to comprise one or more objects that are different from one or more objects in the first video.

20 . The system of claim 13 , wherein the one or more processors are to use one or more neural networks to control the robotic device in an autonomous vehicle.

Assignees

• Nvidia Corp

Inventors

• Mandlekar Ajay Uday
• Nasiriany Soroush
• Wen Bowen
• Akinola Iretiayo
• Narang Yashraj Shyam
• Fan Linxi
• Zhu Yuke
• Fox Dieter

Classifications

• B25J19/023 Primary

  including video camera means · CPC title

• B25J9/1697

  Vision controlled systems · CPC title

• B25J9/1689

  Teleoperation · CPC title

• B25J9/1671

  characterised by simulation, either to verify existing program or to create and verify new program, CAD/CAM oriented, graphic oriented programming systems · CPC title

• B25J9/163

  learning, adaptive, model based, rule based expert control · CPC title