Truck unloader visualization

Patent metadata

Field	Value
Publication number	US-9744669-B2
Application number	US-201514730926-A
Country	US
Kind code	B2
Filing date	Jun 4, 2015
Priority date	Jun 4, 2014
Publication date	Aug 29, 2017
Grant date	Aug 29, 2017

Abstract

Methods, devices, systems, and non-transitory process-readable storage media for a computing device of a robotic carton unloader to identify items to be unloaded from an unloading area within imagery.

First claim

What is claimed is:

1. A method for controlling a robotic carton unloader, comprising:
obtaining first image data from a first sensor of the robotic carton unloader of a carton pile from a first location;
obtaining second image data from a second sensor of the robotic carton unloader of the carton pile from a second location that is different from the first location;
transforming at least one of the first and second image data to be in a consistent global coordinate frame as another of the first and second image data;
combining the transformed first and second image data into combined image data;
detecting cartons in a face of the carton pile in the combined image data from the first and second sensor;
sending indications of the detected cartons; and
controlling a robotic arm of the robotic carton unloader to unload the detected cartons onto a conveyor of the robotic carton unloader based on the indications of the detected cartons.

2. The method of claim 1 , further comprising:
transforming the second image data by deskewing the second image data to have a common reference point as the first image data; and
combining the transformed first and second image data by stitching together the first image data and the deskewed second image data.

3. The method of claim 1 , wherein the first and second sensors are placed at different locations and orientations around the robotic arm and fixed to a platform of the robotic carton unloader and having overlapping views to obtain a complete image of a wall of items at a given time, wherein at least one of the first and second sensors is occluded by the robotic arm in at least one position.

4. The method of claim 1 , wherein controlling the robotic arm of the robotic carton unloader to unload the detected cartons onto the conveyor of the robotic carton unloader based on the indications of the detected cartons comprises controlling the robotic arm of the robotic carton unloader to unload the detected cartons onto the conveyor of the robotic carton unloader based on the indications of the detected cartons without triggering the first sensor to obtain additional image data until all the detected cartons are unloaded.

5. The method of claim 4 , further comprising:
determining that all the detected cartons are unloaded; and
triggering the first sensor to obtain the additional image data of the carton pile to detect cartons in a next face of the carton pile in response to determining that all detected cartons are unloaded.

6. The method of claim 1 , wherein:
obtaining the first image data from the first sensor of the robotic carton unloader of the carton pile comprises obtaining image data from the first sensor of the robotic carton unloader of the carton pile in a first computing device of the robotic carton unloader;
detecting cartons in a face of the carton pile in the first image data from the first sensor comprises detecting cartons in a face of the carton pile in the first image data from the first sensor in the first computing device;
sending indications of the detected cartons comprises sending indications of the detected cartons from the first computing device to a second computing device of the robotic carton unloader, and
controlling the robotic arm of the robotic carton unloader to unload the detected cartons onto a conveyor of the robotic carton unloader based on the indications of the detected cartons comprises controlling, by the second computing device.

7. The method of claim 6 , wherein detecting cartons in the face of the carton pile in the image data from the first sensor in the first computing device comprises performing, by the first computing device, edge detection operations to identify edges of cartons in the face of the carton pile.

8. The method of claim 6 , wherein detecting cartons in the face of the carton pile in the image data from the first sensor in the first computing device comprises detecting one or more of a hole, a label, and a logo in the image data for each detected carton.

9. The method of claim 6 , further comprising:
obtaining second image data from hall the second sensor of the robotic carton unloader of the carton pile in a third computing device of the robotic carton unloader;
detecting cartons in the face of the carton pile in the image data from the second sensor in the third computing device;
obtaining third image data from a third sensor of the robotic carton unloader of the carton pile in a fourth computing device of the robotic carton unloader;
detecting cartons in the face of the carton pile in the third image data from the third sensor in the fourth computing device;
sending indications of the detected cartons from the third computing device and the fourth computing device to the first computing device; and
combining the indications of the detected cartons from the first computing device, the third computing device, and the fourth computing device together to generate a master set of indications of detected cartons at the first computing device, wherein sending indications of the detected cartons from the first computing device to the second computing device of the robotic carton unloader comprises sending the master set of indications of detected cartons from the first computing device to the second computing device of the robotic carton unloader.

10. The method of claim 9 , wherein the first image data from the first sensor, the second image data from the second sensor, and the third image data from the third sensor are each red, green, blue (RGB) data and depth data.

11. The method of claim 10 , wherein the first sensor, the second sensor, and the third sensors are Microsoft Kinect sensors.

12. A robotic carton unloader, comprising:
a vision system, comprising:
a first sensor positioned at a first location;
a second sensor positioned at a second location positioned at a second location that is different from the first location and
a first processor connected to the first sensor,
a conveyor;
a robotic arm; and
a second processor connected to the robotic arm and the first processor of the vision system, wherein the first processor of the vision system is configured with processor executable instructions to perform operations comprising:
obtaining first image data from the first sensor of a carton pile;
obtaining second image data from the second sensor of the carton pile;
transforming at least one of the first and second image data to be in a consistent global coordinate frame as another of the first and second image data;
combining the transformed first and second image data into combined image data;
detecting cartons in a face of the carton pile in the combined image data from the first sensor; and
sending indications of the detected cartons to the second processor, and wherein the second processor is configured with processor executable instructions to perform operations comprising controlling the robotic arm to unload the detected cartons onto the conveyor based on the received indications of the detected cartons.

13. The robotic carton unloader of claim 12 , wherein the first processor of the vision system is configured with processor executable instructions to perform operations such that detecting cartons in the face of the carton pile in the image data from the first sensor comprises performing edge detection operations to identify edges of cartons in the face of the carton pile.

14. The robotic carton unloader of claim 12 , wherein the first processor of the vision system is configured with processor executable instructions to perform operations such that detecting cartons in the face of the carton pil

Assignees

• Intelligrated Headquarters Llc

Inventors

• Wicks Matthew R
• Nilson Benjamin
• Yuvaraj Karthikeyan
• Tankasala Sriram Pavan

Classifications

• Y10S901/47

  Optical · CPC title

• G05B2219/45056

  Handling cases, boxes · CPC title

• B25J9/1679 Primary

  characterised by the tasks executed · CPC title

• B25J9/1697

  Vision controlled systems · CPC title

• B25J9/1664

  characterised by motion, path, trajectory planning · CPC title