Simulating physical interactions for automated systems

Patent metadata

Field	Value
Publication number	US-12403589-B2
Application number	US-202218148548-A
Country	US
Kind code	B2
Filing date	Dec 30, 2022
Priority date	Mar 20, 2022
Publication date	Sep 2, 2025
Grant date	Sep 2, 2025

Abstract

A system may simulate human motion for human-robot interactions, such as may involve a handover of an object. Motion capture can be performed for a hand grasping and moving an object to a location and orientation appropriate for a handover, without a need for a robot to be present or an actual handover to occur. This motion data can be used to separately model the hand and the object for use in a handover simulation, where a component such as a physics engine may be used to ensure realistic modeling of the motion or behavior. During a simulation, a robot control model or algorithm can predict an optimal location and orientation to grasp an object, and an optimal path to move to that location and orientation, using a control model or algorithm trained, based at least in part, using the motion models for the hand and object.

First claim

What is claimed is:

1. A method, comprising:
capturing motion data representing motion of a human hand interacting with an object in a physical environment;
analyzing the motion data using a physics simulator to generate a first representation of the motion of the human hand and a second representation of a motion of the object;
providing, based at least on the first representation and the second representation, a simulation of a virtual hand moving according to the first representation of motion and a virtual object moving according to the second representation of motion in a virtual environment;
detecting, during the simulation, simulated contact of a virtual robot with the virtual object;
deactivating, based at least on detecting the simulated contact, a controller driving the motion of the object in the simulation; and
causing the virtual hand to release a grasp of the virtual object in the virtual environment, wherein subsequent motion of the virtual object in the virtual environment will be dependent at least on operation of the virtual robot and a simulated gravity.

2. The method of claim 1 , further comprising:
causing the motion of the virtual object to be controlled independently of a simulated motion of the virtual hand in the virtual environment; and
ensuring, during the simulated motion, a realistic positioning of the virtual object relative to the virtual hand.

3. The method of claim 1 , wherein the simulation includes simulating one or more collisions between the virtual hand and the virtual object.

4. The method of claim 1 , further comprising:
simulating the gravity in the virtual environment, wherein the virtual object will be caused to fall after release of the virtual object by the virtual hand if the virtual robot does not properly grasp and hold the virtual object in the virtual environment.

5. The method of claim 1 , further comprising:
comparing a simulated handover action performed by the virtual robot, in the virtual environment, against ground truth data for the handover action; and
adjusting one or more parameters of a control model for the virtual robot based at least on the comparing.

6. The method of claim 1 , further comprising:
performing the simulation for multiple models of the virtual hand with respect to multiple models of the virtual object, wherein the multiple models of the virtual object correspond to different types of objects.

7. The method of claim 1 , wherein the motion of the human hand interacting with the object in the physical environment includes at least a motion of the human hand to grasp the object and move the object into a position and an orientation at which a handover is able to occur with respect to a gripper of a physical robot.

8. The method of claim 1 , further comprising:
delaying the causing of the virtual hand to release the grasp of the virtual object until simulated contact of the virtual robot with the virtual object occurs for at least a threshold duration of time.

9. The method of claim 1 , further comprising:
verifying that the simulated contact of the virtual robot with the virtual object corresponds to a permissible type of contact before causing of the virtual hand to release the grasp of the virtual object.

10. A method, comprising:
obtaining motion data representing related motion of a first object and a second object in a physical environment;
analyzing the motion data to generate one or more representations of the related motion;
providing, based at least on the one or more representations, a simulation of the related motion of the first object and the second object in a virtual environment;
detecting interaction of a virtual model of a robot with the first object in the virtual environment; and
modifying simulation of at least one of the first object or the second object in the virtual environment based at least on the interaction.

11. The method of claim 10 , wherein the first object is a physical object, and the second object is a human hand holding the physical object, wherein the interaction of the virtual model of the robot with the first object in the virtual environment includes contact with the first object as part of a handover action.

12. The method of claim 11 , wherein the motion data corresponds to at least a motion of the human hand to grasp the physical object and move the physical object into a position and an orientation at which the handover action is able to occur with respect to an appendage of the robot capable of receiving the object.

13. The method of claim 10 , further comprising:
causing the simulated motion of the first object to be controlled independently of a simulated motion of the second object in the virtual environment; and
ensuring, during the simulated motion, a realistic positioning of the first object relative to the second object.

14. The method of claim 10 , wherein the simulation includes simulating one or more collisions between the first object and the second object.

15. The method of claim 10 , further comprising:
simulating a force of gravity in the virtual environment, wherein the first object is able to be simulated to fall in case of a failed interaction of the virtual model of the robot with the first object.

16. A system, comprising:
one or more processors to:
analyze motion data representing motion of a human hand interacting with an object in a physical environment to generate a first representation of the motion of the human hand and a second representation of corresponding motion of the object;
provide a simulation including a virtual robot to take possession of a virtual object from a virtual hand in a virtual environment;
cause, during the simulation, the virtual hand to move according to the first representation of motion and the virtual object to move according to the second representation of motion; and
determine a successful transfer of possession of the virtual object from the virtual hand to the virtual robot in response to the virtual object being in simulated contact with an end effector of the virtual robot for at least a minimum period of time.

17. The system of claim 16 , wherein the first object is a physical object and the transfer of possession of the virtual object from the virtual hand to the virtual robot is part of a handover action.

18. The system of claim 17 , wherein the motion data corresponds to at least a motion of the human hand to grasp the physical object and move the physical object into a position and an orientation at which the handover action is able to occur with respect to a gripper of the robot.

19. The system of claim 16 , wherein the one or more processors are further to:
cause a simulated motion of the virtual object to be controlled separately from a simulated motion of the virtual hand in the virtual environment; and
ensure a realistic positioning of the virtual object relative to the virtual hand.

20. The system of claim 16 , wherein the system is to:
perform simulation operations;
perform simulation operations to test or validate autonomous machine applications;
render graphical output;
perform deep learning operations;
use an edge device;
incorporate one or more Virtual Machines (VMs);
be implemented at least partially in a data center; or
be implemented at least partially using cloud computing resources.

Assignees

• Nvidia Corp

Inventors

• Chao Yu-Wei
• Xiang Yu
• Yang Wei
• Fox Dieter
• Paxton Chris
• Sundaralingam Balakumar
• Cakmak Maya

Classifications

• B25J9/163

  learning, adaptive, model based, rule based expert control · CPC title

• G05B2219/39001

  Robot, manipulator control · CPC title

• G05B2219/40309

  Simulation of human hand motion · CPC title

• G05B2219/40202

  Human robot coexistence · CPC title

• B25J9/1605 Primary

  Simulation of manipulator lay-out, design, modelling of manipulator · CPC title