Predicting grasp outcomes

Patent metadata

Field	Value
Publication number	US-12420420-B2
Application number	US-202318208752-A
Country	US
Kind code	B2
Filing date	Jun 12, 2023
Priority date	Jun 17, 2022
Publication date	Sep 23, 2025
Grant date	Sep 23, 2025

Abstract

Apparatuses, systems, and techniques to generate a predicted outcome of an object resulting from a robotic component applying a force. In at least one embodiment, a predicted outcome of an object resulting from a robotic component applying a force is generated based on, for example, a neural network.

First claim

What is claimed is:

1. A system, comprising:
at least one processor; and
at least one memory comprising instructions that, in response to execution by the at least one processor, cause the system to at least:
obtain a representation of a grasp pose, corresponding to a mesh of an object, a mesh of a robotic component, and an indication of a force;
generate a refined grasp pose by at least:
using a graph neural network (GNN) to predict at least one of a stress field or a deformation field using the representation; and
generating a modified grasp pose by modifying the grasp pose in a direction indicated by a gradient of an objective function that is based at least in part on the at least one of the stress field or the deformation field; and

cause the robotic component to use the refined grasp pose to apply the force to the object.

2. The system of claim 1 , wherein the objective function calculates a backwards pass-differentiable measure of the at least one of the stress field or the deformation field.

3. The system of claim 1 , wherein obtaining the representation of the grasp pose comprises obtaining a plurality of representations of a plurality of grasp poses, the plurality of representations comprising the representation of the grasp pose; and
using the GNN to predict the at least one of the stress field or the deformation field using the representation comprises using the GNN to predict a plurality of outcomes using the plurality of representations, the plurality of outcomes to comprise at least one of stress fields or deformation fields, which comprise the at least one of the stress field or the deformation field, and
generating the modified grasp pose comprises using output of the objective function calculated for the plurality of outcomes to select the representation representing the grasp pose before modifying the grasp pose.

4. The system of claim 1 , wherein the GNN is differentiable.

5. The system of claim 1 , wherein the at least one memory comprises further instructions that, in response to execution by the at least one processor, cause the system to at least:
obtain training data based, at least in part, on one or more simulators, wherein the training data comprises at least one of at least one training stress field or at least one training deformation fields corresponding to one or more objects; and
train the GNN using at least the training data.

6. The system of claim 1 , wherein the representation is a multigraph.

7. A method, comprising:
obtaining data representing a grasp pose, the data comprising a set of meshes and an indication of a force, wherein the set of meshes are associated with an object and a robotic component;
generating a representation of the data;
using a graph neural network to predict at least one of a stress field or a deformation field using the representation;
refining the grasp pose based at least in part on a gradient of an objective function that is based at least in part on the at least one of the stress field or the deformation field; and
causing the robotic component to use the refined grasp pose to apply the force to the object.

8. The method of claim 7 , wherein the objective function calculates a backwards pass-differentiable measure of the at least one of the stress field or the deformation field.

9. The method of claim 7 , wherein the representation comprises a scalar elastic modulus of the object.

10. The method of claim 7 , further comprising:
causing the graph neural network to process one or more feature vectors associated with the representation in connection with one or more multilayer perceptrons (MLPs) to predict the at least one of the stress field or the deformation field.

11. The method of claim 7 , wherein the object is a deformable object.

12. The method of claim 7 , wherein obtaining the representation of the data comprises obtaining a plurality of representations of information representing a plurality of grasp poses, the plurality of representations comprising the representation of the data; and
using the graph neural network to predict the at least one of the stress field or the deformation field using the representation comprises using the graph neural network to predict a plurality of outcomes using the plurality of representations, the plurality of outcomes to comprise at least one of stress fields or deformation fields comprising the at least one of the stress field or the deformation field, and
the method further comprises using output of the objective function calculated for the plurality of outcomes to select the representation representing the data before refining the grasp pose.

13. A non-transitory computer-readable medium comprising instructions that, when performed by at least one processor of a computing device, cause the computing device to at least:
obtain a representation of a grasp pose, corresponding to mesh of an object, a mesh of a robotic component, and an indication of a force;
use a graph neural network to predict at least one of a stress field or a deformation field using the representation;
generate a modified grasp pose based at least in part on a gradient of an objective function that is based at least in part on the at least one of the stress field or the deformation field; and
cause the robotic component to perform a grasp on the object based at least in part on the modified grasp pose.

14. The non-transitory computer-readable medium of claim 13 , wherein the representation is a multigraph comprising a scalar elastic modulus of the object.

15. The non-transitory computer-readable medium of claim 13 , wherein the robotic component is a gripper; and
the representation is associated with at least a unit vector in a gripper closing direction.

16. The non-transitory computer-readable medium of claim 13 , wherein obtaining the representation of the grasp pose comprises obtaining a plurality of representations of a plurality of grasp poses, the plurality of representations comprising the representation of the grasp pose; and
using the graph neural network to predict the at least one of the stress field or the deformation field using the representation comprises using the graph neural network to predict a plurality of outcomes using the plurality of representations, the plurality of outcomes to comprise at least one of stress fields or deformation fields comprising the at least one of the stress field or the deformation field, and
generating the modified grasp pose comprises using output of the objective function calculated for the plurality of outcomes to select the representation representing the grasp pose.

17. The non-transitory computer-readable medium of claim 13 , comprising further instructions that when performed by the at least one processor of the computing device, cause the computing device to at least:
obtain data from one or more simulation frameworks; and
train the graph neural network using the data in connection with a mean squared error (MSE) based loss.

18. The non-transitory computer-readable medium of claim 13 , wherein the representation is associated with at least one or more normalized grasp forces.

19. The non-transitory computer-readable medium of claim 13 , comprising further instructions that when performed by the at least one processor of the computing device, cause the computing device to at least:
cause one or more processes of the graph neural network to be performed in connection with one or more graphics processing units (GPUs).

20. The non-transitory compute

Assignees

• Nvidia Corp

Inventors

• Huang Isabella
• Narang Yashraj
• Hermans Tucker Ryer
• Tozeto Ramos Fabio
• Handa Ankur
• Macklin Miles Andrew
• Fox Dieter

Classifications

• B25J9/163

  learning, adaptive, model based, rule based expert control · CPC title

• G05B2219/39484

  Locate, reach and grasp, visual guided grasping · CPC title

• B25J9/1671 Primary

  characterised by simulation, either to verify existing program or to create and verify new program, CAD/CAM oriented, graphic oriented programming systems · CPC title

• B25J9/1612 Primary

  characterised by the hand, wrist, grip control · CPC title