Path dataset generation

Patent metadata

Field	Value
Publication number	US-12466065-B2
Application number	US-202418636345-A
Country	US
Kind code	B2
Filing date	Apr 16, 2024
Priority date	Apr 16, 2024
Publication date	Nov 11, 2025
Grant date	Nov 11, 2025

Abstract

A method for generating a dataset of robot motion programs for training a path generation neural network. A large language model is used to configure a task environment and generate code which creates robot simulations. The large language model uses a robot task library and an asset geometry database as inputs. Based on the task and asset inputs and a task instruction, the large language model breaks down the task into steps, then generates code describing robot and object motion to complete the task. The generated code produces robot motions for the task, and a corresponding robot motion program is created and executed in simulation. The simulated robot motion programs are used to generate collision-free robot paths via RRT and/or optimization, and collision-free paths are validated for robot reachability and object placement success. Validated motion programs are added to the dataset and used for training the path generation neural network.

First claim

What is claimed is:

1 . A method for robot path dataset generation, said method comprising:
providing a task library, an object asset library and a textual instruction for a task to a large language model (LLM) running on a computer having a processor and memory; generating code containing programming instructions for a robot to perform the task, by the LLM; executing a simulation of the robot performing the task using the code; generating a collision-free robot path from the simulation using a collision avoidance path generation algorithm; validating the robot path against a set of path quality criteria; and when the robot path meets the path quality criteria, adding the robot path and data defining an obstacle environment to a path dataset.

2 . The method according to claim 1 wherein the task library includes definitions of primitive tasks and task sub-combinations which are combinable to perform the task.

3 . The method according to claim 1 wherein the object asset library includes three-dimensional models of objects involved in the task, including workpieces, robot arm components, grippers, fixtures and obstacles.

4 . The method according to claim 1 wherein generating code includes writing programming instructions defining motions of a tool center point, at an end of a robot arm, necessary to perform the task.

5 . The method according to claim 4 wherein the LLM first writes a narrative of steps necessary to perform the task based on the textual instruction, then writes the programming instructions corresponding with the narrative of steps.

6 . The method according to claim 4 wherein executing a simulation includes calculating motions of all parts of the robot corresponding with the motions of the tool center point.

7 . The method according to claim 6 wherein calculating motions of all parts of the robot includes using an inverse kinematic calculation algorithm.

8 . The method according to claim 1 wherein generating a collision-free robot path includes using the simulation of the robot as an initial path, and using either a rapidly-exploring random tree (RRT) algorithm or an optimization-based algorithm to generate the collision-free robot path.

9 . The method according to claim 1 wherein validating the robot path includes verifying that the robot path is collision-free, that all motions of the robot in the robot path are feasible, and that the task is completed successfully.

10 . The method according to claim 1 wherein the path dataset is populated with a plurality of validated robot paths, each generated based on a different combination of the textual instruction, start and goal points, and the obstacle environment.

11 . The method according to claim 10 further comprising training a neural network system using the path dataset, including supervised learning training of a neural network.

12 . The method according to claim 11 further comprising generating a collision-free robot motion program based on inputs for an operation, using the neural network system, and sending instructions causing a robot to perform the operation using the collision-free robot motion program.

13 . A method for controlling a robot, said method comprising:
generating a collision-free robot motion program based on inputs for an operation, using a neural network system running on a computing device, and sending instructions causing the robot to perform the operation using the collision-free robot motion program, where the neural network system is trained using a path dataset in a supervised learning process, and where the path dataset is populated with a plurality of validated paths, each path generated based on a different combination of a textual instruction, start and goal points, and an obstacle environment, using steps including; providing a task library, an object asset library and the textual instruction for a task to a large language model (LLM); generating code containing programming instructions for the robot to perform the task, using the LLM; executing a simulation of the robot performing the task using the code; generating a collision-free robot path from the simulation using a collision avoidance path generation algorithm; validating the robot path against a set of path quality criteria; and when the robot path meets the path quality criteria, adding the robot path and data defining the obstacle environment to the path dataset.

14 . A system for robot path dataset generation, said system comprising:
a computer having a processor and memory, said computer running a large language model (LLM) and an algorithm configured with steps including; providing a task library, an object asset library and a textual instruction for a task to running on; generating code containing programming instructions for a robot to perform a task, by the LLM, using a task library, an object asset library and a textual instruction for the task; executing a simulation of the robot performing the task using the code; generating a collision-free robot path from the simulation using a collision avoidance path generation algorithm; validating the robot path against a set of path quality criteria; and when the robot path meets the path quality criteria, adding the robot path and data defining an obstacle environment to a path dataset.

15 . The system according to claim 14 wherein the task library includes definitions of primitive tasks and task sub-combinations which are combinable to perform the task, and the object asset library includes three-dimensional models of objects involved in the task, including workpieces, robot arm components, grippers, fixtures and obstacles.

16 . The system according to claim 14 wherein generating code includes writing programming instructions defining motions of a tool center point, at an end of a robot arm, necessary to perform the task, and where the LLM first writes a narrative of steps necessary to perform the task based on the textual instruction, then writes the programming instructions corresponding with the narrative of steps.

17 . The system according to claim 16 wherein executing a simulation includes calculating motions of all parts of the robot corresponding with the motions of the tool center point.

18 . The system according to claim 14 wherein generating a collision-free robot path includes using the simulation of the robot as an initial path, and using either a rapidly-exploring random tree (RRT) algorithm or an optimization-based algorithm to generate the collision-free robot path.

19 . The system according to claim 14 wherein validating the robot path includes verifying that the robot path is collision-free, that all motions of the robot in the robot path are feasible, and that the task is completed successfully.

20 . The system according to claim 14 wherein the path dataset is populated with a plurality of validated robot paths, each generated based on a different combination of the textual instruction, start and goal points, and the obstacle environment.

21 . The system according to claim 20 wherein the path dataset is used to train a neural network, including supervised learning training of a neural network.

22 . The system according to claim 21 further comprising a robot in communication with a robot controller, where a collision-free robot motion program is generated based on inputs for an operation, using the neural network system, and the robot controller sends instructions causing th

Assignees

• Fanuc Corp

Inventors

• Wang Kaimeng
• Kato Tetsuaki

Classifications

• G06N3/0895

  Weakly supervised learning, e.g. semi-supervised or self-supervised learning · CPC title

• G06N5/041

  Abduction · CPC title

• G06N3/0455

  Auto-encoder networks; Encoder-decoder networks · CPC title

• B25J9/161

  Hardware, e.g. neural networks, fuzzy logic, interfaces, processor · CPC title

• B25J9/1676 Primary

  Avoiding collision or forbidden zones · CPC title