Vision calibration system for robotic carton unloading

Patent metadata

Field	Value
Publication number	US-11049282-B2
Application number	US-201916288953-A
Country	US
Kind code	B2
Filing date	Feb 28, 2019
Priority date	Feb 28, 2019
Publication date	Jun 29, 2021
Grant date	Jun 29, 2021

Abstract

The present disclosure relates to a method and system for calibrating a carton detection system. The method includes receiving a three-dimensional (3D) point cloud of a calibration object, determining a 3D target pose of the calibration object by comparing the 3D point cloud to a point cloud template, receiving a two-dimensional (2D) optical image of the calibration object, identifying one or more markers of the calibration object based on the 2D optical image, determining a marker pose for each of the one or more markers based on the 2D optical image, determining a 2D target pose based on the marker pose for each of the one or more markers, generating a transformation matrix based on the 3D target pose and the 2D target pose, and calibrating the carton detection system based on the transformation matrix.

First claim

What is claimed is:

1. A method of calibrating a carton detection system, the method comprising:
receiving a three-dimensional (3D) point cloud of a calibration object;
determining a 3D target pose of the calibration object by comparing the 3D point cloud of the calibration object to a point cloud template;
receiving a two-dimensional (2D) optical image of the calibration object;
identifying one or more markers of the calibration object based on the 2D optical image of the calibration object;
determining a marker pose for each of the one or more markers of the calibration object based on the 2D optical image;
determining a 2D target pose based on the marker pose for each of the one or more markers of the calibration object;
generating a transformation matrix based on the 3D target pose and the 2D target pose; and
calibrating the carton detection system based on the transformation matrix, wherein calibrating the carton detection system comprises projecting one or more points of the 3D point cloud on the 2D optical image based on the transformation matrix.

2. The method of claim 1 , wherein identifying the one or more markers of the calibration object comprises detecting one or more edges and corners of the one or more markers from the 2D optical image.

3. The method of claim 1 , wherein determining the marker pose for each of the one or more markers comprises estimating the marker pose based on at least one of a camera focus and marker size.

4. The method of claim 1 , wherein determining the 2D target pose based on the marker pose further comprises determining a marker location for each of the one or more markers.

5. The method of claim 1 , wherein generating the transformation matrix further comprises:
generating a 3D_to_target transformation matrix based on the 3D target pose;
generating a 2D_to_target transformation matrix based on the 2D target pose; and
generating the transformation matrix based on the 3D_to_target transformation matrix and the 2D_to_target transformation matrix.

6. The method of claim 1 , wherein generating the transformation matrix comprises generating a rotation submatrix and a translation submatrix.

7. The method of claim 6 , further comprising generating the rotation submatrix based on one or more of a roll angle, a pitch angle, or a yaw angle.

8. The method of claim 6 , further comprising generating the translation submatrix based on a translation between one or more of the 3D point cloud, the 2D optical image, and the point cloud template.

9. A robotic carton handling system for unloading cartons, the robotic carton handling system comprising:
a mobile body;
a movable robotic manipulator attached to the mobile body, wherein the movable robotic manipulator comprises an end effector configured to unload one or more cartons from a carton pile; and
a carton detection system comprising:
one or more sensors coupled to at least one of the mobile body or the movable robotic manipulator, wherein the one or more sensors are configured to provide a two-dimensional (2D) optical image and a three-dimensional (3D) point cloud of a calibration object; and
a processing subsystem in communication with the one or more sensors, wherein the processing subsystem is configured to:
receive the 3D point cloud of the calibration object from the one or more sensors;
determine a 3D target pose of the calibration object by comparing the 3D point cloud of the calibration object to a point cloud template;
receive the 2D optical image of the calibration object;
identify one or more markers of the calibration object based on the 2D optical image of the calibration object;
determine a marker pose for each of the one or more markers of the calibration object based on the 2D optical image;
determine a 2D target pose based on the marker pose for each of the one or more markers of the calibration object;
generate a transformation matrix based on the 3D target pose and the 2D target pose;
calibrate the carton detection system based on the transformation matrix; and
project one or more points of the 3D point cloud on the 2D optical image based on the transformation matrix.

10. The robotic carton handling system of claim 9 , wherein the one or more sensors generate the 2D optical image and the 3D point cloud of an I-shaped target defining one or more markers.

11. The robotic carton handling system of claim 9 , wherein the one or more sensors further configured to generate data related to a shape of the calibration object.

12. The robotic carton handling system of claim 9 , wherein the processing subsystem is further configured to detect one or more edges and corners of the one or more markers from the 2D optical image.

13. The robotic carton handling system of claim 9 , wherein the processing subsystem is further configured to estimate the marker pose based on at least one of a camera focus and a marker size.

14. The robotic carton handling system of claim 9 , wherein the processing subsystem, in generating the transformation matrix, is further configured to generate a rotation submatrix and a translation submatrix.

15. The robotic carton handling system of claim 9 , wherein the processing subsystem is further configured to:
generate a 3D_to_target transformation matrix based on the 3D target pose;
generate a 2D_to_target transformation matrix based on the 2D target pose; and
generate the transformation matrix based on the 3D_to_target transformation matrix and the 2D_to_target transformation matrix.

16. A material handling system comprising:
a robotic carton handling system for unloading cartons in a carton pile, the robotic carton handling system, the robotic carton handling system comprising:
a mobile body;
a movable robotic manipulator attached to the mobile body, wherein the movable robotic manipulator comprises an end effector configured to unload one or more cartons from the carton pile;
a calibration object; and
a carton detection system comprising:
one or more sensors coupled to at least one of the mobile body or the movable robotic manipulator and configured to generate a two-dimensional (2D) optical image and a three-dimensional (3D) point cloud of the calibration object; and
a processing subsystem in communication with the one or more sensors, the processing subsystem configured to:
receive the 3D point cloud of the calibration object from the one or more sensors;
determine a 3D target pose of the calibration object by comparing the 3D point cloud of the calibration object to a point cloud template;
receive the 2D optical image of the calibration object;
identify one or more markers on the calibration object based on the 2D optical image of the calibration object;
determine a marker pose for each of the one or more markers on the calibration object based on the 2D optical image;
determine a 2D target pose based on the marker pose for each of the one or more markers on the calibration object;
generate a transformation matrix based on the 3D target pose and the 2D target pose;
calibrate the carton detection system based on the transformation matrix; and
project one or more points of the 3D point cloud on the 2D optical image based on the transformation matrix.

17. The material handling system of claim 16 , wherein the calibration object comprises an I-shaped target defining one or more markers.

18. The material handling system of claim 16 , wherein the one or more sensors is further configured to generate data related to a shape of the calibration object.

Assignees

• Intelligrated Headquarters Llc

Inventors

• Zhang Feng
• Tankasala Sriram Pavan

Classifications

• G06T7/70

  Determining position or orientation of objects or cameras (camera calibration G06T7/80) · CPC title

• B25J9/1697

  Vision controlled systems · CPC title

• G06T7/85 Primary

  Stereo camera calibration · CPC title

• B65G1/1373

  for fulfilling orders in warehouses · CPC title

• G06T2207/10028

  Range image; Depth image; 3D point clouds · CPC title