Generating a grasp pose for grasping of an object by a grasping end effector of a robot

Patent metadata

Field	Value
Publication number	US-9987744-B2
Application number	US-201715592849-A
Country	US
Kind code	B2
Filing date	May 11, 2017
Priority date	May 11, 2016
Publication date	Jun 5, 2018
Grant date	Jun 5, 2018

Abstract

Generating a grasp pose for grasping of an object by an end effector of a robot. An image that captures at least a portion of the object is provided to a user via a user interface output device of a computing device. The user may select one or more pixels in the image via a user interface input device of the computing device. The selected pixel(s) are utilized to select one or more particular 3D points that correspond to a surface of the object in the robot's environment. A grasp pose is determined based on the particular 3D points. For example, a local plane may be fit based on the particular 3D point(s) and a grasp pose determined based on a normal of the local plane. Control commands can be provided to cause the grasping end effector to be adjusted to the grasp pose, after which a grasp is attempted.

First claim

What is claimed is:

1. A method implemented by one or more processors, comprising:
providing, for display on a client computing device, an image capturing at least a portion of an object in an environment, the image being generated based on image data generated by one or more cameras that are associated with a robot and that are viewing the environment;
receiving selection data in response to providing the image, the selection data indicating one or more first selected pixels of the image and one or more second selected pixels of the image,
the first selected pixels being selected based on a first selection by a user via a user interface input device of the client computing device, and
the second selected pixels being selected based on a second selection by the user via the user interface input device;

selecting, based on the first selected pixels and the second selected pixels, at least one particular three-dimensional (3D) point from a group of 3D points, wherein the group of 3D points is generated based on the image data generated by the one or more cameras or determined based on additional sensor data generated by a 3D scanner that is associated with the robot and that is viewing the environment;
determining a grasp approach vector based on the particular 3D point;
determining a grasp pose that conforms to the grasp approach vector; and
providing one or more control commands that cause a grasping end effector of the robot to traverse to the grasp pose and attempt a grasp of the object subsequent to traversing to the grasp pose.

2. The method of claim 1 , wherein the particular 3D point is not mapped directly to the first selected pixels and is not mapped directly to the second selected pixels.

3. The method of claim 2 , wherein the first selection and the second selection are opposed to one another and the particular 3D point is between, in at least one dimension, one or more first 3D points mapped directly to the first selected pixels and one or more second 3D points mapped directly to the second selected pixels.

4. The method of claim 3 , wherein selecting the particular 3D point based on the first selected pixels and the second selected pixels comprises:
determining an additional pixel that is between, in the image, the first selected pixels and the second selected pixels; and
selecting the particular 3D point based on the particular 3D point being mapped directly to the additional pixel.

5. The method of claim 4 , wherein determining the grasp pose further comprises:
determining the grasp pose based on the one or more first 3D points that are mapped directly to the first selected pixels, and based on the one or more second 3D points that are mapped directly to the second selected pixels.

6. The method of claim 5 , wherein determining the grasp pose based on the one or more first 3D points and the one or more second 3D points comprises:
projecting the one or more first 3D points and the one or more second 3D points to coordinates in a local plane determined based on the particular 3D point; and
determining the grasp pose based on conformance of one or more actuable members of the grasping end effector to the coordinates in the local plane.

7. The method of claim 1 , wherein determining the grasp pose comprises:
generating a plurality of candidate grasp poses that each conform to the grasp approach vector; and
selecting the grasp pose from the candidate grasp poses based on:
the grasp pose avoiding collision with the object and with other objects in the environment, or
one or more additional criteria.

8. The method of claim 1 , wherein the selection data further indicates one or more third selected pixels selected based on a third selection by the user via the user interface input device, and wherein selecting the particular 3D point is further based on the third selected pixels.

9. A method implemented by one or more processors, comprising:
providing, for display on a client computing device, a two-dimensional (2D) image capturing at least a portion of an object in an environment, the 2D image being generated based on image data generated by one or more cameras, of a robot, that are viewing the environment;
receiving selection data in response to providing the 2D image, the selection data indicating one or more selected pixels of the 2D image, the one or more selected pixels being selected based on at least one selection of the 2D image via a user interface input device of the client computing device;
selecting, based on the one or more selected pixels indicated by the selection data, at least one particular three-dimensional (3D) point from a group of 3D points, the group of 3D points being generated based on the image data generated by the cameras or determined based on additional sensor data generated by a 3D scanner, of the robot, that is viewing the environment;
determining a grasp approach vector based on the particular 3D point;
determining a grasp pose that conforms to the grasp approach vector; and
providing one or more control commands that cause a grasping end effector of the robot to traverse to the grasp pose and attempt a grasp of the object subsequent to traversing to the grasp pose.

10. The method of claim 9 , wherein determining the grasp pose that conforms to the grasp approach vector comprises:
employing a collision checker using the grasp approach vector as a constraint and using, as obstacles, additional 3D points of the group of 3D points.

11. The method of claim 9 , wherein the one or more selected pixels comprise a plurality of pixels selected based on at least two selections via the user interface input device.

12. The method of claim 11 , wherein the particular 3D point is not mapped directly to any of the selected pixels.

13. The method of claim 11 , wherein the two selections are opposed to one another and the particular 3D point is between, in at least one dimension, other 3D points of the group of 3D points, the other 3D points being mapped directly to the selected pixels.

14. The method of claim 13 , wherein selecting the particular 3D points based on the selected pixels comprises:
determining an additional pixel that is between, in the 2D image, the selected pixels of the two selections; and
selecting the particular 3D point based on the particular 3D point being mapped directly to the additional pixel.

15. The method of claim 13 , wherein determining the grasp pose further comprises:
determining the grasp pose based on the other 3D points that are mapped directly to the selected pixels.

16. The method of claim 9 , wherein determining the grasp pose comprises:
generating a plurality of candidate grasp poses that each conform to the grasp approach vector; and
selecting the grasp pose from the candidate grasp poses based on:
the grasp pose avoiding collision with the object and with other objects in the environment, or
one or more additional criteria.

17. A system, comprising:
a camera viewing an environment of a robot;
at least one processor configured to:
provide, for display on a client computing device, a two-dimensional (2D) image capturing at least a portion of an object in an environment, the 2D image being generated based on image data generated by the camera;
receive selection data in response to providing the 2D image, the selection data indicating one or more selected pixels of the 2D image, the one or more selected pixels being selected based on at least one selection of the 2D image via a user interface input device of the client computing device;
select, based on the one or m

Assignees

• X Dev Llc

Inventors

• Prats Mario

Classifications

• H04N13/0203

  Electricity · mapped topic

• G06F3/04815

  Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object · CPC title

• G06K9/00805

  Physics · mapped topic

• G06K9/481

  Physics · mapped topic

• B25J19/021

  Optical sensing devices · CPC title