Target-free RGBD camera alignment to robots

Patent metadata

Field	Value
Publication number	US-11724407-B2
Application number	US-202117151453-A
Country	US
Kind code	B2
Filing date	Jan 18, 2021
Priority date	Jan 18, 2021
Publication date	Aug 15, 2023
Grant date	Aug 15, 2023

Abstract

One embodiment provides a robotic system comprising: a robot, the robot further comprising a moveable robotic arm that moves within the robot's reference space; a depth-sensing camera, the camera having a reference frame that is in substantial view of the robot's reference space; a controller, the controller further comprising a processor and a computer readable memory that comprises instructions such that, when read by the controller, the controller inputs image data from the camera and sends signals to the moveable robotic arm, the instructions further comprising the steps of: calibrating the camera to the robot by instructing the robot to engage in a number of robot poses; extracting the location of the robot poses to obtain the robot poses in the camera reference frame; and creating a transformation that transforms robot points afterwards to camera points.

First claim

The invention claimed is:

1. In a robotic system whose motions are autonomously controlled via images captured by a depth-sensing camera system, a method for determining camera parameters relative to a robot, the steps comprising:
calibrating the camera to the robot by instructing the robot to engage in a number of robot poses;
extracting the location of the robot poses to obtain the robot poses in the camera reference frame; and
creating a transformation that transforms robot points afterwards to camera points.

2. The method of claim 1 wherein calibrating the camera to the robot further comprises sending instructions to the robot to pose at points in 3D space that are not collinear.

3. The method of claim 1 wherein calibrating the camera to the robot further comprises sending instructions to the robot to pose at points in 3D space that are not coplanar.

4. The method of claim 1 wherein calibrating the camera to the robot further comprises sending instructions to the robot to pose at points in 3D space that are substantially occupy the robot work space.

5. The method of claim 1 wherein calibrating the camera to the robot further comprises sending instructions to the robot to pose at least 8 points in 3D space that substantially form a cube in the robot work space.

6. The method of claim 1 wherein calibrating the camera to the robot further comprises obtaining a set of 3D points as follows:
X R =[X R1 ,X R2 , . . . ,X Rn ],
where each point X Ri has three dimensions, X Ri(1) , X Ri(2) , X Ri(3) .

7. The method of claim 1 wherein extracting the location of the robot poses to obtain the robot poses in the camera reference frame further comprises localizing a target on the robot in an image frame within a specific window.

8. The method of claim 7 wherein localizing a target on the robot in an image frame within a specific window further comprises detecting a known object on the tip of the robot tool.

9. The method of claim 7 wherein localizing a target on the robot in an image frame within a specific window further comprises identifying a desired color blob on the tip of the robot tool.

10. The method of claim 7 wherein extracting the location of the robot poses to obtain the robot poses in the camera reference frame further comprises finding the centroid of the largest connected component of the robot tool.

11. The method of claim 10 wherein extracting the location of the robot poses to obtain the robot poses in the camera reference frame further comprises sampling the depths of the target from the depth-sensing camera around the centroid.

12. The method of claim 11 wherein extracting the location of the robot poses to obtain the robot poses in the camera reference frame further comprises generating a set of points in the camera frame as follows:
X C =[X C1 ,X C2 , . . . ,X Cn ].

13. The method of claim 1 wherein creating a transformation that transforms robot points afterwards to camera points further comprises finding a transformation of robot points X R =[X R1 , X R2 , . . . , X Rn ] to camera points X C =[X C1 , X C2 , . . . , X Cn ].

14. The method of claim 13 wherein creating a transformation that transforms robot points afterwards to camera points further comprises finding the relative position of the two point sets using epipolar constraints.

15. The method of claim 14 wherein creating a transformation that transforms robot points afterwards to camera points further comprises expressing the robot points and camera points in homogenous coordinates.

16. The method of claim 15 wherein creating a transformation that transforms robot points afterwards to camera points further comprises finding the fundamental/essential matrices from the homogenous coordinates.

17. A robotic system comprising:
a robot, the robot further comprising a moveable robotic arm that moves within the robot's reference space;
a depth-sensing camera, the camera having a reference frame that is in substantial view of the robot's reference space;
a controller, the controller further comprising a processor and a computer readable memory that comprises instructions such that, when read by the controller, the controller inputs image data from the camera and sends signals to the moveable robotic arm, the instructions further comprising the steps of:
calibrating the camera to the robot by instructing the robot to engage in a number of robot poses;
extracting the location of the robot poses to obtain the robot poses in the camera reference frame; and
creating a transformation that transforms robot points afterwards to camera points.

18. The robotic system of claim 17 wherein the moveable robotic arm comprise one of a group, the group comprising: a known detectable object on the tip of the robot arm and a desired color blob on the tip of the robot arm.

19. The robotic system of claim 17 wherein the controller finds a transformation of robot points X R =[X R1 , X R2 , . . . , X Rn ] to camera points X C =[X C1 , X C2 , . . . , X Cn ].

20. The robotic system of claim 19 wherein the controller creates a transformation that transforms robot points afterwards to camera points further comprises finding the fundamental/essential matrices from the homogenous coordinates.

Assignees

• Palo Alto Res Ct Inc
• Xerox Corp

Inventors

• Price Robert
• Evans Kent

Classifications

• B25J19/023 Primary

  including video camera means · CPC title

• G06T7/80 Primary

  Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration · CPC title

• G06T2207/30208

  Marker matrix · CPC title

• G06T2207/30244

  Camera pose · CPC title

• G06T7/74

  involving reference images or patches · CPC title