Reactive interactions for robotic applications and other automated systems

Patent metadata

Field	Value
Publication number	US-12454050-B2
Application number	US-202217854730-A
Country	US
Kind code	B2
Filing date	Jun 30, 2022
Priority date	Mar 20, 2022
Publication date	Oct 28, 2025
Grant date	Oct 28, 2025

Abstract

Approaches presented herein provide for predictive control of a robot or automated assembly in performing a specific task. A task to be performed may depend on the location and orientation of the robot performing that task. A predictive control system can determine a state of a physical environment at each of a series of time steps, and can select an appropriate location and orientation at each of those time steps. At individual time steps, an optimization process can determine a sequence of future motions or accelerations to be taken that comply with one or more constraints on that motion. For example, at individual time steps, a respective action in the sequence may be performed, then another motion sequence predicted for a next time step, which can help drive robot motion based upon predicted future motion and allow for quick reactions.

First claim

What is claimed is:

1. A method, comprising:
receiving data representative of an environment, the environment including an object held by a human hand;
determining, based at least in part on evaluating one or more potential grasp options corresponding to a robot, a target grasp option from the one or more potential grasp options;
determining a sequence of motions between an initial position and a final position corresponding to the robot according to the target grasp option, the sequence of motions being determined based at least in part on minimizing one or more cost functions and satisfying one or more motion constraints;
causing, at individual time steps corresponding to one or more motions of the sequence of motions, the robot to perform a respective motion of the sequence of motions;
detecting contact of the robot with the object corresponding to the target grasp position; and
causing an end-effector of the robot to grasp the object.

2. The method of claim 1 , wherein the one or more motion constraints include at least one of a constraint to favor smooth motion, a constraint to limit acceleration, a constraint to favor straight line motion, a constraint to avoid a collision, a constraint to avoid a self-collision, or a constraint to avoid an occlusion of a sensor used to capture the data.

3. The method of claim 1 , wherein the determining the sequence of motions comprises using a model predictive control (MPC) system to execute at least one optimization algorithm with the one or more motion constraints.

4. The method of claim 3 , wherein the determining the target grasp option is executed using the MPC system.

5. The method of claim 3 , wherein the MPC system is to optimize the sequence of motions over individual potential grasp options of the one or more potential grasp options.

6. The method of claim 1 , further comprising:
modifying, based at least in part on one or more user inputs, the one or more motion constraints.

7. The method of claim 1 , further comprising:
monitoring, during the individual time steps corresponding to the one or more motions of the sequence of motions, whether the end-effector contacts the human hand; and
upon a determination that the end-effector has contacted the human hand, performing one or more operations.

8. The method of claim 1 , wherein individual motions of the sequence of motions are determined using one or more joint accelerations optimized for the individual time steps.

9. A method, comprising:
determining, during a sequence of time steps, a set of positions for a robot to perform an action;
determining, between a current position and a final position of the set of positions, a sequence of motions that satisfy one or more motion constraints, the sequence of motions allowing for a change in a target position from among the set of positions at individual time steps of the sequence of time steps;
causing, at individual time steps of the sequence of time steps, the robot to perform a respective motion of the sequence of motions in order to move at least a portion of the robot with respect to the target position; and
causing the robot to perform the action when at least the portion of the robot is determined to be within a threshold distance to the target position.

10. The method of claim 9 , wherein one or more positions of the set of positions include location and orientation information.

11. The method of claim 9 , wherein the sequence of motions is determined using a predictive model and one or more optimization criteria.

12. The method of claim 11 , wherein the predictive model optimizes the sequence of motions over the set of positions and evaluates the target position at the individual time steps of the sequence of time steps.

13. The method of claim 9 , wherein the one or more motion constraints include at least one of a constraint to favor smooth motion, a constraint to limit acceleration, a constraint to favor straight line motion, a constraint to avoid a collision, a constraint to avoid a self-collision, or a constraint to avoid an occlusion of a sensor.

14. The method of claim 9 , further comprising:
capturing, during the sequence of time steps, sensor data representative of a physical environment in which the robot is to perform the action, and
determining the sequence of motions based at least in part on a change determined with respect to the physical environment.

15. A system, comprising:
one or more processing units to:
determine, during a sequence of time steps, a set of positions for a robot to perform an action;
determine, between a current position and a final position of the set of positions, a sequence of motions that satisfy one or more motion constraints, the sequence of motions allowing for a change in a target position from among the set of positions at individual time steps of the sequence of time steps;
cause, at individual time steps of the sequence of time steps, the robot to perform a respective motion of the sequence of motions in order to move at least a portion of the robot with respect to the target position; and
cause the robot to perform the action when at least the portion of the robot is determined to be within a threshold distance to the target position.

16. The system of claim 15 , wherein the one or more processing units are further to:
use a predictive model to determine the sequence of motions based, at least in part, on one or more optimization criteria for the sequence of motions.

17. The system of claim 16 , wherein the one or more processing units are further to:
optimize, using the predictive model, the sequence of motions over the set of positions and evaluate the target position at the individual time steps of the sequence of time steps.

18. The system of claim 15 , wherein the one or more processing units are further to:
capture, during the sequence of time steps, sensor data representative of a physical environment in which the robot is to perform the action, and
determine the change in target position based at least in part on a change determined with respect to the physical environment.

19. The system of claim 15 , wherein the one or more motion constraints include at least one of a constraint to favor smooth motion, a constraint to limit acceleration, a constraint to favor straight line motion, a constraint to avoid a collision, a constraint to avoid a self-collision, or a constraint to avoid an occlusion of a sensor.

20. The system of claim 15 , wherein the system comprises at least one of:
a control system for an autonomous or semi-autonomous machine;
a perception system for an autonomous or semi-autonomous machine;
a system for performing simulation operations;
a system for performing digital twin operations;
a system for performing light transport simulation;
a system for performing collaborative content creation for 3D assets;
a system for performing deep learning operations;
a system implemented using an edge device;
a system implemented using a robot;
a system for performing conversational AI operations;
a system for generating synthetic data;
a system incorporating one or more virtual machines (VMs);
a system implemented at least partially in a data center; or
a system implemented at least partially using cloud computing resources.

Assignees

• Nvidia Corp

Inventors

• Yang Wei
• Sundaralingam Balakumar
• Paxton Christopher Jason
• Cakmak Maya
• Chao Yu-Wei
• Fox Dieter
• Akinola Iretiayo

Classifications

• B25J9/1666

  Avoiding collision or forbidden zones · CPC title

• G05B2219/50391

  Robot · CPC title

• B25J9/1605

  Simulation of manipulator lay-out, design, modelling of manipulator · CPC title

• G05B2219/40269

  Naturally compliant robot arm · CPC title

• G05B19/4155

  characterised by program execution, i.e. part program or machine function execution, e.g. selection of a program · CPC title