Simulation driven robotic control of real robot(s)

Patent metadata

Field	Value
Publication number	US-12472630-B2
Application number	US-202418582266-A
Country	US
Kind code	B2
Filing date	Feb 20, 2024
Priority date	Dec 31, 2020
Publication date	Nov 18, 2025
Grant date	Nov 18, 2025

Abstract

Active utilization of a robotic simulator in control of one or more real world robots. A simulated environment of the robotic simulator can be configured to reflect a real world environment in which a real robot is currently disposed, or will be disposed. The robotic simulator can then be used to determine a sequence of robotic actions for use by the real world robot(s) in performing at least part of a robotic task. The sequence of robotic actions can be applied, to a simulated robot of the robotic simulator, to generate a sequence of anticipated simulated state data instances. The real robot can be controlled to implement the sequence of robotic actions. The implementation of one or more of the robotic actions can be contingent on a real state data instance having at least a threshold degree of similarity to a corresponding one of the anticipated simulated state data instances.

First claim

What is claimed is:

1 . A method implemented by one or more processors, the method comprising:
configuring a simulated environment, of a robotic simulator that simulates the simulated environment and a simulated robot, based on real environment state data that is based on one or more sensor-based observations of a real environment; determining, using the robotic simulator, a first candidate sequence of robotic actions and a second candidate sequence of robotic actions, for use in controlling a real robot, in the real environment, in performing at least part of a robotic task; applying, to the simulated robot of the robotic simulator and with the simulated environment after the configuring, the first candidate sequence of robotic actions to generate first simulated state data; applying, to the simulated robot of the robotic simulator and with the simulated environment after the configuring, the second candidate sequence of robotic actions to generate second simulated state data; determining, based on the first simulated state data and the second simulated state data, to utilize the first candidate sequence of robotic actions, in lieu of the second candidate sequence of robotic actions, for use in controlling the real robot in performing at least part of the robotic task; determining, prior to causing the real robot to implement the first candidate sequence of robotic actions and based on comparing a portion of the simulated environment to a corresponding portion of updated real environment state data, that a degree of similarity between the portion of the simulated environment and the corresponding portion of the updated real environment state data satisfies a threshold; and in response to determining to utilize the first candidate sequence of robotic actions and based on the degree of similarity between the portion of the simulated environment and the corresponding portion of the updated real environment state data satisfying the threshold:
causing the real robot to implement the first candidate sequence of robotic actions.

2 . The method of claim 1 , wherein determining, based on the first simulated state data and the second simulated state data, to utilize the first candidate sequence of robotic actions, in lieu of the second candidate sequence of robotic actions, comprises:
generating one or more first features based on the first simulated data; generating one or more second features based on the second simulated data; and determining to utilize the first candidate sequence of robotic actions based on comparing the first features to the second features.

3 . The method of claim 2 , wherein the one or more first features include a first task success feature and a first efficiency feature, and wherein the one or more second features include a second task success feature and a second efficiency feature.

4 . The method of claim 2 , wherein the one or more first features include a first task success feature and the one or more second features include a second task success feature.

5 . The method of claim 2 , wherein the one or more first features include a first efficiency feature, and wherein the one or more second features include a second efficiency feature.

6 . The method of claim 1 , wherein the first candidate sequence of robotic actions indicates a corresponding sequence of control commands to be implemented at corresponding time steps.

7 . The method of claim 6 , wherein in implementing the first candidate sequence of robotic actions the real robot sends the control commands, indicated by the first candidate sequence of robotic actions, to corresponding actuators of the real robot.

8 . A method implemented by one or more processors, the method comprising:
configuring a simulated environment, of a robotic simulator that simulates the simulated environment, based on real environment state data that is based on one or more sensor-based observations of a real environment; determining, using the robotic simulator, a candidate sequence of robotic actions for use in controlling a real robot, in the real environment, in performing at least part of a robotic task, wherein determining the candidate sequence of robotic actions comprises:
processing simulated state data that simulates real state data that is fully incapable of being generated utilizing any sensors in the real environment, wherein the simulated state data is generated from a simulated viewpoint that corresponds to a real viewpoint, in the real environment, and wherein none of the sensors in the real environment are capable of, without human intervention, capturing real vision data from the real viewpoint, in the real environment, that corresponds to the simulated viewpoint from which the simulated state data is generated; and
causing the real robot to implement the candidate sequence of robotic actions.

9 . The method of claim 8 , wherein the simulated viewpoint is an overhead viewpoint.

10 . The method of claim 8 , wherein the simulated state data simulates the real state data that is fully incapable of being generated utilizing any sensors in the real environment by being generated from a simulated sensor of a particular type and wherein none of any sensors in the real environment are of the particular type.

11 . The method of claim 8 , wherein the candidate sequence of robotic actions indicates a corresponding sequence of control commands to be implemented at corresponding time steps.

12 . A system, comprising:
memory storing instructions; one or more processors executing the instructions to:
configure a simulated environment, of a robotic simulator that simulates the simulated environment and a simulated robot, based on real environment state data that is based on one or more sensor-based observations of a real environment;
determine, using the robotic simulator, a first candidate sequence of robotic actions and a second candidate sequence of robotic actions, for use in controlling a real robot, in the real environment, in performing at least part of a robotic task;
apply, to the simulated robot of the robotic simulator and with the simulated environment after the configuring, the first candidate sequence of robotic actions to generate first simulated state data;
apply, to the simulated robot of the robotic simulator and with the simulated environment after the configuring, the second candidate sequence of robotic actions to generate second simulated state data; determine, based on the first simulated state data and the second simulated state data, to utilize the first candidate sequence of robotic actions, in lieu of the second candidate sequence of robotic actions, for use in controlling the real robot in performing at least part of the robotic task; determine, prior to causing the real robot to implement the first candidate sequence of robotic actions and based on comparing a portion of the simulated environment to a corresponding portion of updated real environment state data, that a degree of similarity between the portion of the simulated environment and the corresponding portion of the updated real environment state data satisfies a threshold; and in response to determining to utilize the first candidate sequence of robotic actions and based on the degree of similarity between the portion of the simulated environment and the corresponding portion of the updated real environment state data satisfying the threshold:
cause the real robot to implement the first candidate sequence of robotic actions.

13 . The system of claim 12 , wherein in determining, based on the first simulated state data and the second simulated state data, to utilize th

Assignees

• Gdm Holding Llc

Inventors

• Bai Yunfei
• Gasparian Tigran
• Austin Brent
• Christiansen Andreas
• Bennice Matthew
• Bechard Paul

Classifications

• B25J9/1697

  Vision controlled systems · CPC title

• B25J9/1661

  characterised by task planning, object-oriented languages · CPC title

• G05B2219/32385

  What is simulated, manufacturing process and compare results with real process · CPC title

• B25J9/1671 Primary

  characterised by simulation, either to verify existing program or to create and verify new program, CAD/CAM oriented, graphic oriented programming systems · CPC title

• G05B17/02 Primary

  electric · CPC title