3D-2D vision system for robotic carton unloading

Patent metadata

Field	Value
Publication number	US-10315866-B2
Application number	US-201715481969-A
Country	US
Kind code	B2
Filing date	Apr 7, 2017
Priority date	Oct 20, 2016
Publication date	Jun 11, 2019
Grant date	Jun 11, 2019

Abstract

Robotic carton loader or unloader incorporates three-dimensional (3D) and two-dimensional (2D) sensors to detect respectively a 3D point cloud and a 2D image of a carton pile within transportation carrier such as a truck trailer or shipping container. Edge detection is performed using the 3D point cloud, discarding segments that are two small to be part of a product such as a carton. Segments that are too large to correspond to a carton are 2D image processed to detect additional edges. Results from 3D and 2D edge detection are converted in a calibrated 3D space of the material carton loader or unloader to perform one of loading or unloading of the transportation carrier. Image processing can also detect jamming of products sequence from individually controllable zones of a conveyor of the robotic carton loader or unloader for singulated unloading.

First claim

What is claimed is:

1. A method of determining locations of individual cartons in a material handling system, the method comprising:
receiving, by a carton detection system, a two-dimensional (2D) image and a three-dimensional (3D) point cloud of at least one portion of a carton pile resting on a floor of a transportation carrier;
detecting, by a processing subsystem in connection with the carton detection system, segments within the 3D point cloud;
removing, by the processing subsystem, any segments that are smaller than a first threshold;
determining, by the processing subsystem, whether any segments are less than a second threshold;
in response to determining that a selected segment is less than the second threshold, qualifying the selected segment as a 3D detected carton;
in response to determining that a selected segment is not less than the second threshold:
determining, by the processing subsystem, a 2D mask that corresponds to the selected segment;
determining, by the processing subsystem, a portion of the 2D image that corresponds to the 2D mask;
detecting, by the processing subsystem, segments within the portion of the 2D image; and
qualifying, by the processing subsystem, detected segments as 2D detected cartons;

combining, by the processing subsystem, the 2D and 3D detected cartons in a detection result; and
converting, by the processing subsystem, the detection result using calibration information into 3D locations relative to a robotic carton handling system for a selected one of a loading operation and an unloading operation performed by a robotic carton manipulator of the robotic carton handling system.

2. The method of claim 1 , further comprising:
receiving the 2D image and 3D point cloud from one or more sensors positioned on the robotic carton handling system to detect the one portion of the carton pile;
receiving a 2D image and 3D point cloud from another one or more sensors positioned on the robotic carton handling system to detect a contiguous portion of the carton pile; and
combining the 2D and 3D detected cartons from both the one portion and the contiguous portion to form the detection result.

3. The method of claim 1 , wherein qualifying detected segments as 2D detected cartons further comprises discarding, by the processing subsystem, edges that form a smaller rectangle fully encompassed within a larger rectangle.

4. The method of claim 1 , further comprising:
performing the selected one of the loading operation and the unloading operation by the robotic carton handling system using the 3D locations.

5. A robotic carton handling system for unloading cartons in a carton pile, the robotic carton handling system movable across a floor, the robotic carton handling system comprising:
a mobile body;
a movable robotic manipulator attached to the mobile body and comprising an end effector at an end thereof, the end effector configured to unload one or more cartons from the carton pile;
a conveyor mounted on the mobile body configured to receive the one or more cartons from the end effector and to move the one or more cartons towards a rear of the robotic carton handling system;
a carton detection system comprising:
one or more sensors coupled respectively to one of the mobile body and the movable robotic manipulator to provide a two-dimensional (2D) optical image and a three-dimensional (3D) point cloud of at least one portion of a carton pile resting on a floor of a transportation carrier;
a processing subsystem in communication with the one or more sensors, the processing subsystem:
detects segments within the 3D point cloud;
removes any segments that are smaller than a first threshold;
determines whether any segments are less than a second threshold;
in response to determining that a selected segment is less than the second threshold, qualifies the selected segment as a 3D detected carton;
in response to determining that a selected segment is not less than the second threshold:
determines a 2D mask that corresponds to the selected segment;
determines a portion of the 2D optical image that corresponds to the 2D mask;
detects segments within the portion of the 2D optical image; and
qualifies detected segments as 2D detected cartons;

combines the 2D and 3D detected cartons in a detection result; and
converts the detection result using calibration information into 3D locations relative to a robotic carton handling system for a selected one of a loading operation and an unloading operation.

6. The robotic carton handling system of claim 5 , wherein:
the one or more sensors are positioned on the robotic carton handling system to detect the one portion of the carton pile;
the robotic carton handling system further comprises another one or more sensors positioned on the robotic carton handling system to detect a contiguous portion of the carton pile; and
the processor subsystem combines the 2D and 3D detected cartons from both the one portion and the contiguous portion to form the detection result.

7. The robotic carton handling system of claim 5 , wherein the processor subsystem qualifies detected segments as 2D detected cartons at least in part by discarding edges that form a smaller rectangle fully encompassed within a larger rectangle.

8. The robotic carton handling system of claim 5 , further comprising an automation controller in communication with the processing subsystem, the automation controller causes the robotic carton manipulator to perform the selected one of the loading operation and the unloading operation by the robotic carton handling system using the 3D locations.

9. A material handling system comprising:
a robotic carton handling system for unloading cartons in a carton pile, the robotic carton handling system movable across a floor, the robotic carton handling system comprising:
a mobile body;
a movable robotic manipulator attached to the mobile body and comprising an end effector at an end thereof, the end effector configured to unload one or more cartons from the carton pile;
a conveyor mounted on the mobile body configured to receive the one or more cartons from the end effector and to move the one or more cartons towards a rear of the robotic carton handling system;
a carton detection system comprising:
one or more sensors coupled respectively to one of the mobile body and the movable robotic manipulator to provide a two-dimensional (2D) optical image and a three-dimensional (3D) point cloud of at least one portion of a carton pile resting on a floor of a transportation carrier;
a processing subsystem in communication with the one or more sensors, the processing subsystem:
detects segments within the 3D point cloud;
removes any segments that are smaller than a first threshold;
determines whether any segments are less than a second threshold;
in response to determining that a selected segment is less than the second threshold, qualifies the selected segment as a 3D detected carton;
in response to determining that a selected segment is not less than the second threshold:
determines a 2D mask that corresponds to the selected segment;
determines a portion of the 2D optical image that corresponds to the 2D mask;
detects segments within the portion of the 2D optical image; and qualifies detected segments as 2D detected cartons;

combines the 2D and 3D detected cartons in a detection result; and
converts the detection result using calibration information into 3D locations relative to a robotic carton handling system for a selected one of a loading operation and an unloading operation;

an automation controller in communication with the processing subsystem, the automation controller causes the robotic carton manipulator to perf

Assignees

• Intelligrated Headquarters Llc

Inventors

• Yuvaraj Karthikeyan
• Tankasala Sriram Pavan
• Nilson Benjamin E
• Zhang Feng
• Wicks Matthew R

Classifications

• G05B2219/45056

  Handling cases, boxes · CPC title

• G05B2219/40442

  Voxel map, 3-D grid map · CPC title

• G05B2219/40006

  Placing, palletize, un palletize, paper roll placing, box stacking · CPC title

• B25J9/1697

  Vision controlled systems · CPC title

• B25J9/1687

  Assembly, peg and hole, palletising, straight line, weaving pattern movement · CPC title