Method and system for performing automatic camera calibration for robot control

Patent metadata

Field	Value
Publication number	US-10369698-B1
Application number	US-201916295940-A
Country	US
Kind code	B1
Filing date	Mar 7, 2019
Priority date	Mar 7, 2019
Publication date	Aug 6, 2019
Grant date	Aug 6, 2019

Abstract

A robot control system and a method for automatic camera calibration is presented. The robot control system includes a control circuit configured to determine all corner locations of an imaginary cube that fits within a camera field of view, and determine a plurality of locations that are distributed on or throughout the imaginary cube. The control circuit is further configured to control a robot arm to move a calibration pattern to the plurality of locations, and to receive a plurality of calibration images corresponding to the plurality of locations, and to determine respective estimates of intrinsic camera parameters based on the plurality of calibration images, and to determine an estimate of a transformation function that describes a relationship between a camera coordinate system and a world coordinate system. The control circuit is further configured to control placement of the robot arm based on the estimate of the transformation function.

First claim

What is claimed is:

1. A robot control system comprising:
a communication interface configured to communicate with a robot and with a camera having a camera field of view, wherein the robot has a base and a robot arm with a calibration pattern disposed thereon; and
a control circuit configured to perform camera calibration by:
determining all corner locations of an imaginary cube that fits within the camera field of view,
determining a plurality of locations that are distributed on or throughout the imaginary cube,
controlling the robot arm to move the calibration pattern to the plurality of locations that are distributed on or throughout the imaginary cube by outputting movement commands to the robot via the communication interface,
receiving a plurality of calibration images from the camera via the communication interface, wherein the plurality of calibration images are captured by the camera, and are respective images of the calibration pattern at the plurality of locations,
determining respective estimates of intrinsic camera parameters based on the plurality of calibration images, and
determining, based on the respective estimates of the intrinsic camera parameters: an estimate of a transformation function that describes a relationship between a camera coordinate system and a world coordinate system, wherein the camera coordinate system is a coordinate system defined with respect to a location and orientation of the camera, and the world coordinate system is a coordinate system defined with respect to a location that is stationary relative to the base of the robot,

wherein the control circuit is further configured, after the camera calibration is performed, to receive a subsequent image from the camera via the communication interface, and to control placement of the robot arm by outputting to the robot, via the communication interface, a subsequent movement command that is based on the subsequent image and based on the estimate of the transformation function.

2. The robot control system of claim 1 , wherein the plurality of locations are uniformly distributed on or throughout the imaginary cube.

3. The robot control system of claim 2 , wherein the plurality of locations have uniform spacing, as measured along an edge of the imaginary cube, between immediately neighboring locations of the plurality of locations.

4. The robot control system of claim 1 , wherein the control circuit is configured to control the robot arm to move the calibration pattern to the plurality of locations in response to a determination that the robot arm is able to move the calibration pattern to all corner locations of the imaginary cube.

5. The robot control system of claim 4 , wherein the control circuit is configured to control the robot arm to move the calibration pattern to the plurality of locations only in response to: (a) the determination that the robot arm is able to move the calibration pattern to all corner locations of the imaginary cube, and (b) a determination that the robot arm is able, at each corner location of all corner locations of the imaginary cube, to tilt the calibration pattern to an angle that is within a defined range of angles relative to the camera.

6. The robot control system of claim 5 , wherein the imaginary cube is a second imaginary cube determined by the control circuit, and wherein the control circuit is further configured
to determine a first imaginary cube that fits within the camera field of view,
to determine that the robot arm is not able to move the calibration pattern to one or more corner locations of the first imaginary cube, or that the robot arm is not able, for one or more corner locations of the first imaginary cube, to tilt the calibration pattern to an angle that is within the defined range of angles relative to the camera,
wherein the control circuit is configured to determine all corner locations of the second imaginary cube in response to at least one of: (a) a determination that the robot arm is not able to move the calibration pattern to one or more corner locations of the first imaginary cube, or (b) the robot arm is not able, for one or more corner locations of the first imaginary cube, to tilt the calibration pattern to an angle that is within the defined range of angles relative to the camera.

7. The robot control system of claim 6 , wherein the first imaginary cube is a maximum-sized imaginary cube that is able to fit within the camera field of view, and wherein the second imaginary cube is smaller than the first imaginary cube.

8. The robot control system of claim 1 , wherein the plurality of locations include exactly n 3 locations, wherein n is an integer that is equal to or greater than 2.

9. The robot control system of claim 1 , wherein the control circuit is configured to control the robot arm, via the movement commands, to tilt the calibration pattern to different respective angles relative to the camera for the plurality of locations that are uniformly distributed on or throughout the imaginary cube, such that the plurality of respective calibration images capture the calibration pattern at different respective angles relative to the camera.

10. The robot control system of claim 1 , wherein the calibration pattern includes a plurality of pattern elements, and wherein the control circuit is configured,
to receive a first calibration image from the camera before the plurality of calibration images are received, wherein the first calibration image is an image of the calibration pattern and is captured by the camera before the plurality of respective calibration images are captured,
to determine a level of intensity and a level of contrast of at least one of the pattern elements in the first calibration image, and
to determine respective values of an exposure parameter and a focus parameter of the camera based on at least one of the level of intensity and the level of contrast of at least one of the pattern elements in the first calibration image,
wherein the plurality of respective calibration images are captured by the camera with the respective values of the exposure parameter and the focus parameter.

11. The robot control system of claim 1 , wherein the control circuit is configured to determine the plurality of locations by dividing the imaginary cube into a plurality of non-overlapping regions, and by assigning a respective location of the plurality of locations to be in each region of the plurality of non-overlapping regions.

12. The robot control system of claim 1 , wherein the control circuit is configured to determine the plurality of locations over a series of iterations, wherein a first location of the plurality of locations is determined as any location within the imaginary cube, and is determined during a first iteration of the series of iterations, wherein the imaginary cube forms a first region used to perform the first iteration, and
wherein respective locations for the remaining iterations are determined by performing the following for each of the remaining iterations: (a) dividing a region used to perform a previous iteration into a first half region and a second half region that do not overlap with each other, wherein each of the first half region and the second half region is a region used to perform a current iteration, (b) determining which of the first half region and the second half region contains a previous location, wherein the previous location is a respective location of the plurality of locations determined in the previous iteration, (c) and determining any location within the other of first half region and the second half region as a current location, wherein the current location is a location of

Assignees

• Mujin Inc

Inventors

• Islam Russell
• Ye Xutao
• Diankov Rosen

Classifications

• G06T1/0014

  Image feed-back for automatic industrial control, e.g. robot with camera (robots B25J19/023) · CPC title

• G05B2219/39016

  Simultaneous calibration of manipulator and camera · CPC title

• G05B2219/39008

  Fixed camera detects reference pattern held by end effector · CPC title

• G05B2219/37015

  Adaptive online camera, vision calibration · CPC title

• B25J9/1697 Primary

  Vision controlled systems · CPC title