Simulation driven robotic control of real robot(s)

Patent metadata

Field	Value
Publication number	US-11938638-B2
Application number	US-202117337815-A
Country	US
Kind code	B2
Filing date	Jun 3, 2021
Priority date	Dec 31, 2020
Publication date	Mar 26, 2024
Grant date	Mar 26, 2024

Abstract

Active utilization of a robotic simulator in control of one or more real world robots. A simulated environment of the robotic simulator can be configured to reflect a real world environment in which a real robot is currently disposed, or will be disposed. The robotic simulator can then be used to determine a sequence of robotic actions for use by the real world robot(s) in performing at least part of a robotic task. The sequence of robotic actions can be applied, to a simulated robot of the robotic simulator, to generate a sequence of anticipated simulated state data instances. The real robot can be controlled to implement the sequence of robotic actions. The implementation of one or more of the robotic actions can be contingent on a real state data instance having at least a threshold degree of similarity to a corresponding one of the anticipated simulated state data instances.

First claim

What is claimed is:

1. A method implemented by one or more processors, the method comprising:
configuring a simulated environment, of a robotic simulator, based on real environment state data that is based on one or more sensor-based observations of a real environment;
determining, using the robotic simulator, a sequence of robotic actions for use in controlling a real robot, in the real environment, in performing at least part of a robotic task;
applying, to a simulated robot of the robotic simulator, the sequence of robotic actions to generate a sequence of simulated state data instances, wherein each of the simulated state data instances defines, for a corresponding point in the sequence of robotic actions one or both of:
corresponding anticipated environmental state data for the corresponding point, and
corresponding anticipated robot state data for the corresponding point;
subsequent to determining the sequence of robotic actions:
controlling the real robot to implement a first subset of the sequence of robotic actions;
comparing, during or after controlling the real robot to implement the first subset of the sequence of robotic actions, a real state data instance to a corresponding one of the simulated state data instances for the corresponding point in the sequence of robotic actions, the real state data instance defining, for the corresponding point, one or both of:
updated real environment state data, that is based on one or more updated sensor-based observations of the real environment, and
real robot state data that is based on one or more sensor-based observations of the real robot; and
in response to the comparing indicating a threshold degree of similarity between the updated real state data and the corresponding one of the anticipated state data instances for the corresponding point:
controlling the real robot to implement a second subset of the sequence of robotic actions,
wherein controlling the real robot to fully implement the second subset of the sequence of robotic actions is contingent on the comparing indicating the threshold degree of similarity between the updated real state data and the corresponding one of the anticipated state data instances for the corresponding point.

2. The method of claim 1 , further comprising:
in response to the comparing not indicating the threshold degree of similarity:
halting controlling of the real robot based on the sequence of robotic actions.

3. The method of claim 2 , further comprising, in response to the comparing not indicating the threshold degree of similarity:
reconfiguring the simulated environment based on updated real environment state data;
determining, using the robotic simulator, an updated sequence of robotic actions for use in controlling the real robot in performing the robotic task;
applying, to the simulated robot of the robotic simulator, the updated sequence of robotic actions to generate an updated sequence of simulated state data instances, wherein each of the updated simulated state data instances defines, for a corresponding updated point in the updated sequence of robotic actions, one or both of:
corresponding updated anticipated environmental state data for the corresponding updated point, and
real robot state data that is based on one or more sensor-based observations of the real robot;

controlling the real robot to implement a first updated subset of the updated sequence of robotic actions;
comparing an updated real state data instance to a corresponding updated one of the updated simulated state data instances for the corresponding updated point in the updated sequence of robotic actions;
in response to the comparing, the updated real state data instance to the corresponding updated one of the updated simulated state data instances, indicating the threshold degree of similarity:
controlling the real robot to implement an updated second subset of the updated sequence of robotic actions.

4. The method of claim 1 , wherein the method includes controlling the real robot to implement the second subset of the sequence of robotic actions and to implement any additional subsets of the sequence of robotic actions.

5. The method of claim 4 , further comprising:
prior to full implementation of the sequence of robotic actions:
determining, using the robotic simulator, a next sequence of robotic actions for use in controlling the real robot in performing another part of the robotic task; and
applying, to a simulated robot of the robotic simulator and with the simulated robot and the environment of the robot simulator corresponding to a terminal one of the simulated state data instances, the next sequence of robotic actions to generate a next sequence of simulated state data instances; and

after full implementation of the sequence of robotic actions:
controlling the real robot to implement the next sequence or robotic actions.

6. The method of claim 5 , wherein controlling the real robot to implement the next sequence of robotic actions comprises:
controlling the real robot to implement a first next subset of the next sequence of robotic actions;
comparing a next real state data instance to a corresponding one of the next simulated state data instances;
in response to the comparing indicating the threshold degree of similarity:
controlling the real robot to implement a second next subset of the next sequence of robotic actions.

7. The method of claim 1 , wherein the one or more sensor-based observations comprise robot-sensor-based observations that are based on one or more real sensors of the real robot.

8. The method of claim 1 , wherein the one or more sensor-based observations comprises environment-sensor-based observations that are based on one or more real sensors that are not coupled to the real robot, but are in the real environment.

9. The method of claim 1 , wherein determining the sequence of robotic actions for use in controlling the real robot in performing at least part of the robotic task, comprises:
applying an alternate sequence of robotic actions to the robotic simulator;
comparing:
one or more alternate features, from the applying the alternate sequence of robotic actions to the robotic simulator, to
one or more features, from the applying the sequence of robotic actions to the robotic simulator; and

determining, based on the comparing the one or more alternate features to the one or more features, to utilize the sequence of robotic actions, in lieu of the alternate sequence of robotic actions, for use in controlling the real robot in performing at least part of the robotic task.

10. The method of claim 9 , wherein the one or more features comprise one or both of a task success feature and an efficiency feature, and wherein the one or more alternate features comprise one or both of an alternate task success feature and an alternate efficiency feature.

11. The method of claim 10 , wherein the one or more features comprise the task success feature and further comprising generating the task success feature based on a terminal state, of the robotic simulator, after applying the sequence of robotic actions.

12. The method of claim 10 , wherein the one or more features comprise the efficiency feature, and further comprising generating the efficiency feature based on one or both of:
a quantity of robotic actions in the sequence of robotic actions, and
a distance traversed by one or more simulated components, of the simulated robot, in performing the sequence of robotic actions.

13. The method of claim 1 , wherein comparing the real state data instance to the corresponding one of the simulated state data instance

Assignees

• Google Llc

Inventors

• Bai Yunfei
• Gasparian Tigran
• Austin Brent
• Christiansen Andreas
• Bennice Matthew
• Bechard Paul

Classifications

• B25J9/1671 Primary

  characterised by simulation, either to verify existing program or to create and verify new program, CAD/CAM oriented, graphic oriented programming systems · CPC title

• B25J9/1661

  characterised by task planning, object-oriented languages · CPC title

• B25J9/1697

  Vision controlled systems · CPC title

• G05B2219/32385

  What is simulated, manufacturing process and compare results with real process · CPC title

• G05B2219/32342

  Real time simulation · CPC title