Simulating multiple robots in virtual environments

Patent metadata

Field	Value
Publication number	US-11813748-B2
Application number	US-202017069684-A
Country	US
Kind code	B2
Filing date	Oct 13, 2020
Priority date	Oct 13, 2020
Publication date	Nov 14, 2023
Grant date	Nov 14, 2023

Abstract

Implementations are provided for operably coupling multiple robot controllers to a single virtual environment, e.g., to generate training examples for training machine learning model(s). In various implementations, a virtual environment may be simulated that includes an interactive object and a plurality of robot avatars that are controlled independently and contemporaneously by a corresponding plurality of robot controllers that are external from the virtual environment. Sensor data generated from a perspective of each robot avatar of the plurality of robot avatars may be provided to a corresponding robot controller. Joint commands that cause actuation of one or more joints of each robot avatar may be received from the corresponding robot controller. Joint(s) of each robot avatar may be actuated pursuant to corresponding joint commands. The actuating may cause two or more of the robot avatars to act upon the interactive object in the virtual environment.

First claim

What is claimed is:

1. A method implemented using one or more processors, comprising:
simulating a three-dimensional virtual environment that includes an interactive object, wherein the virtual environment includes a plurality of robot avatars that are controlled independently and contemporaneously by a corresponding plurality of robot controllers that are integral with a plurality of physical robots that are external from the virtual environment and operably coupled with one or more of the processors, wherein the simulating includes generating, for each robot avatar of the plurality of robot avatars, a directed acyclic graph with nodes that represent joints or sensors of the robot avatar;
providing, to each robot controller of the plurality of robot controllers, sensor data that is generated from a perspective of the respective robot avatar of the plurality of robot avatars that is controlled by the robot controller, wherein the sensor data provided to each robot controller is perceived by the robot controller as real-world sensor data;
receiving, from each robot controller of the plurality of robot controllers, joint commands that cause actuation of one or more joints of the respective robot avatar of the plurality of robot avatars that is controlled by the robot controller; and
actuating one or more joints of each robot avatar of the plurality of robot avatars pursuant to corresponding joint commands, wherein the actuating causes two or more of the plurality of robot avatars to act upon the interactive object in the virtual environment.

2. The method of claim 1 , wherein the directed acyclic graph is a dependency graph in which edges between nodes represent dependencies between the joints or sensors represented by the nodes.

3. The method of claim 2 , wherein at least one node representing a simulated sensor of the robot avatar imposes a delay on output of the sensor being passed up the directed acyclic graph, wherein the delay corresponds temporally to an operational frequency of a real-world sensor corresponding to the simulated sensor.

4. The method of claim 2 , wherein one or more nodes of the directed acyclic graph represent a simulated environmental condition of the virtual environment.

5. The method of claim 2 , wherein one or more nodes of the directed acyclic graph represent a simulated condition of a simulated sensor of the robot avatar.

6. The method of claim 1 , wherein the actuating comprises operating the plurality of robot avatars at frequencies that correspond to real-world frequencies of the respective real-world robots in which the robot controllers that control the plurality of robot avatars are integral.

7. The method of claim 6 , further comprising:
operating a simulated world clock of the virtual environment at a given frequency that is greater than or equal to a highest frequency achievable by any of the plurality of robot controllers; and
operating a first robot avatar of the plurality of robot avatars at a first frequency that is less than the given frequency.

8. The method of claim 6 , further comprises pausing a simulated world clock until a robot avatar of the plurality of robot avatars advances to a next stage of operation.

9. A system comprising one or more processors and memory storing instructions that, in response to execution of the instructions by the one or more processors, cause the one or more processors to:
simulate a three-dimensional virtual environment that includes an interactive object, wherein the virtual environment includes a plurality of robot avatars that are controlled independently and contemporaneously by a corresponding plurality of robot controllers that are integral with a plurality of physical robots that are external from the virtual environment and operably coupled with one or more of the processors, wherein the instructions to simulate include instructions to generate, for each robot avatar of the plurality of robot avatars, a directed acyclic graph with nodes that represent joints or sensors of the robot avatar;
provide, to each robot controller of the plurality of robot controllers, sensor data that is generated from a perspective of the respective robot avatar of the plurality of robot avatars that is controlled by the robot controller, wherein the sensor data provided to each robot controller is perceived by the robot controller as real-world sensor data;
receive, from each robot controller of the plurality of robot controllers, joint commands that cause actuation of one or more joints of the respective robot avatar of the plurality of robot avatars that is controlled by the robot controller; and
actuate one or more joints of each robot avatar of the plurality of robot avatars pursuant to corresponding joint commands, wherein the actuating causes two or more of the plurality of robot avatars to act upon the interactive object in the virtual environment.

10. The system of claim 9 , wherein the directed acyclic graph is a dependency graph in which edges between nodes represent dependencies between the joints or sensors represented by the nodes.

11. The system of claim 10 , wherein at least one node representing a simulated sensor of the robot avatar imposes a delay on output of the sensor being passed up the directed acyclic graph, wherein the delay corresponds temporally to an operational frequency of a real-world sensor corresponding to the simulated sensor.

12. At least one non-transitory computer-readable medium comprising instructions that, in response to execution of the instructions by one or more processors, cause the one or more processors to perform the following operations:
simulating a three-dimensional virtual environment that includes an interactive object, wherein the virtual environment includes a plurality of robot avatars that are controlled independently and contemporaneously by a corresponding plurality of robot controllers that are integral with a plurality of physical robots that are external from the virtual environment and operably coupled with one or more of the processors, wherein the instructions to simulate include instructions to generate, for each robot avatar of the plurality of robot avatars, a directed acyclic graph with nodes that represent joints or sensors of the robot avatar;
providing, to each robot controller of the plurality of robot controllers, sensor data that is generated from a perspective of the respective robot avatar of the plurality of robot avatars that is controlled by the robot controller, wherein the sensor data provided to each robot controller is perceived by the robot controller as real-world sensor data;
receiving, from each robot controller of the plurality of robot controllers, joint commands that cause actuation of one or more joints of the respective robot avatar of the plurality of robot avatars that is controlled by the robot controller; and
actuating one or more joints of each robot avatar pursuant to corresponding joint commands, wherein the actuating causes two or more of the plurality of robot avatars to act upon the interactive object in the virtual environment.

Assignees

• Google Llc

Inventors

• Bennice Matthew
• Bechard Paul

Classifications

• B25J9/1605 Primary

  Simulation of manipulator lay-out, design, modelling of manipulator · CPC title

• B25J9/0084

  comprising a plurality of manipulators · CPC title

• B25J9/1671 Primary

  characterised by simulation, either to verify existing program or to create and verify new program, CAD/CAM oriented, graphic oriented programming systems · CPC title

• B25J9/163

  learning, adaptive, model based, rule based expert control · CPC title

• G05B2219/40514

  Computed robot optimized configurations to train ann, output path in real time · CPC title