Robotic vision

Patent metadata

Field	Value
Publication number	US-11433545-B2
Application number	US-202016792009-A
Country	US
Kind code	B2
Filing date	Feb 14, 2020
Priority date	Feb 17, 2019
Publication date	Sep 6, 2022
Grant date	Sep 6, 2022

Abstract

A method includes accessing RGB and depth image data representing a scene that includes at least a portion of a robotic limb. Using this data, a computing system may segment the image data to isolate and identify at least a portion of the robotic limb within the scene. The computing system can determine a current pose of the robotic limb within the scene based on the image data, joint data, or a 3D virtual model of the robotic limb. The computing system may then determine a desired goal pose, which may be based on the image data or the 3D virtual model. Based on the determined goal pose, the computing device determines the difference between the current pose and the goal pose of the robotic limb, and using this difference, provides a pose adjustment that for the robotic limb.

First claim

What is claimed is:

1. A computer-implemented method comprising:
accessing, by a computing device, image data representing a scene including at least a portion of a robotic limb and an object;
segmenting, by the computing device, the image data to isolate at least a portion of the image data representing at least the portion of the robotic limb;
segmenting, by the computing device, the image data to isolate at least a second portion of the image data representing at least one or more areas of the object, wherein the one or more areas of the object are based on preferred interaction areas for the object;
determining, by the computing device, a current pose of the portion of the robotic limb based on at least one or more of:
the first segmented portion of the image data;
joint data from one or more joint encoders of the robotic limb; or
a three-dimensional (3D) virtual model of the robotic limb;

determining, by the computing device, a goal pose of the portion of the robotic limb based at least on one or more of:
the first and second segmented portion of the image data; or
the 3D virtual model of the robotic limb;

determining, by the computing device, a difference between the current pose and the goal pose of the portion of the robotic limb; and
providing, by the computing device and based on the determined difference, a pose adjustment for the robotic limb.

2. The method of claim 1 , comprising:
accessing, by the computing device, one or more measured joint angles of the robotic limb, wherein the measured joint angles are based on the joint data;
determining, by the computing device and based on the one or more measured joint angles, a measured pose of the portion of the robotic limb; and
determining, by the computing device and based on the image data, the current pose of the portion of the robotic limb, wherein the goal pose of the portion of the robotic limb is further based at least on a difference between the current pose of the portion of the robotic limb as determined based on the image data and the measured pose of the portion of the robotic limb as determined based on the measured joint angles.

3. The method of claim 1 , wherein:
the second segmented portion of the image data further isolates the object.

4. The method of claim 1 , wherein the pose adjustment comprises a three-dimensional path for the robotic limb from the current pose to the goal position that avoids the object.

5. The method of claim 1 , further comprising determining, by the computing device and based on the second segmented portion of the image data, a current pose of the object, wherein the goal pose is based on the pose of the object.

6. The method of claim 1 , wherein the goal pose is further based on a physical interaction between the robotic limb and the object.

7. The method of claim 6 , wherein the physical interaction comprises grasping the one or more of the preferred interaction areas of the object.

8. The method of claim 5 , further comprising:
identifying, by the computing device and based on the second segmented portion of the image data, a pose of the one or more areas of the object; and
the goal pose is further based on the pose of the one or more areas of the object.

9. One or more computer-readable non-transitory storage media embodying software that is operable when executed to cause one or more processors to perform operations comprising:
accessing, by a computing device, image data representing a scene including at least a portion of a robotic limb and an object;
segmenting, by the computing device, the image data to isolate at least a portion of the image data representing at least the portion of the robotic limb;
segmenting, by the computing device, the image data to isolate at least a second portion of the image data representing at least one or more areas of the object, wherein the one or more areas of the object are based on preferred interaction areas for the object;
determining, by the computing device, a current pose of the portion of the robotic limb based on at least one or more of:
the first segmented portion of the image data;
joint data from one or more joint encoders of the robotic limb; or
a three-dimensional (3D) virtual model of the robotic limb;

determining, by the computing device, a goal pose of the portion of the robotic limb based at least on one or more of:
the first and second segmented portion of the image data; or
the 3D virtual model of the robotic limb;

determining, by the computing device, a difference between the current pose and the goal pose of the portion of the robotic limb; and
providing, by the computing device and based on the determined difference, a pose adjustment for the robotic limb.

10. The storage media of claim 9 , wherein the operations further comprise:
accessing, by the computing device, one or more measured joint angles of the robotic limb, wherein the measured joint angles are based on the joint data;
determining, by the computing device and based on the one or more measured joint angles, a measured pose of the portion of the robotic limb; and
determining, by the computing device and based on the image data, the current pose of the portion of the robotic limb, wherein the goal pose of the portion of the robotic limb is further based at least on a difference between the current pose of the portion of the robotic limb as determined based on the image data and the measured pose of the portion of the robotic limb as determined based on the measured joint angles.

11. The storage media of claim 9 , wherein:
the second segmented portion of the image data further isolates the object.

12. The storage media of claim 9 , wherein the pose adjustment comprises a three-dimensional path for the robotic limb from the current pose to the goal pose that avoids the object.

13. The storage media of claim 9 , wherein the operations further comprise determining, by the computing device and based on the second segmented portion of the image data, a current pose of the object, wherein the goal pose is based on the pose of the object.

14. The storage media of claim 9 , wherein the goal pose is further based on a physical interaction between the robotic limb and the object.

15. The storage media of claim 13 , wherein the physical interaction comprises grasping the one or more of the preferred interaction areas of the object.

16. The storage media of claim 13 , wherein the operations further comprise:
identifying, by the computing device and based on the second segmented portion of the image data, a pose of the one or more areas of the object; and
the goal pose is further based on the pose of the one or more areas of the object.

17. A system comprising: one or more processors and one or more computer-readable non-transitory storage media coupled to one or more of the processors, the one or more computer-readable non-transitory storage media comprising instructions operable when executed by one or more of the processors to cause the system to perform operations comprising:
accessing, by a computing device, image data representing a scene including at least a portion of a robotic limb and an object;
segmenting, by the computing device, the image data to isolate at least a portion of the image data representing at least the portion of the robotic limb;
segmenting, by the computing device, the image data to isolate at least a second portion of the image data representing at least one or more areas of the object, wherein the one or more areas of the object are based on preferred interaction areas for the o

Assignees

• Samsung Electronics Co Ltd

Inventors

• Schoessler Philipp
• Neveu Thibault
• Mason Dane
• Hajash Kathleen Sofia
• Harms Brian Thomas

Classifications

• B25J9/042

  comprising an articulated arm · CPC title

• G06T7/11

  Region-based segmentation · CPC title

• B25J9/1005

  comprising adjusting means · CPC title

• B25J9/1666

  Avoiding collision or forbidden zones · CPC title

• G06T7/75

  involving models · CPC title