Methods and systems for detecting, recognizing, and localizing pallets

Patent metadata

Field	Value
Publication number	US-10328578-B2
Application number	US-201715494227-A
Country	US
Kind code	B2
Filing date	Apr 21, 2017
Priority date	Apr 21, 2017
Publication date	Jun 25, 2019
Grant date	Jun 25, 2019

Abstract

Example implementations may relate methods and systems for detecting, recognizing, and localizing pallets. For instance, a computing system may receive sensor data representing aspects of an environment, and identify a set of edge points in the sensor data. The computing system may further determine a set of line segments from the set of edge points where each line segment may fit to a subset of the set of edge points. Additionally, the computing system may also filter the set of line segments to exclude line segments that have a length outside a height range and a width range associated with dimensions of a pallet template, and identify, from the filtered set of line segments, a subset of line segments that align with the pallet template. Based on the identified subset of line segments, the computing system may determine a pose of a pallet in the environment.

First claim

What is claimed is:

1. A method comprising:
receiving, at a computing system and from a sensor, sensor data representing aspects of an environment;
identifying a set of edge points in the sensor data, wherein identifying the set of edge points involves: (i) determining a first local surface normal relative to a surface represented in a cluster of data points in the sensor data and a second local surface normal relative to the surface represented in the cluster of data points, and (ii) identifying data points in the cluster of data points that correspond to edge points based on the first local surface normal and the second local surface normal;
determining a set of line segments from the set of edge points, wherein each line segment is fit to a subset of the set of edge points;
filtering the set of line segments to exclude line segments that have a length outside a height range and a width range associated with dimensions of a pallet template;
identifying, from the filtered set of line segments, a subset of line segments that align with the pallet template; and
determining a pose of a pallet in the environment based on the identified subset of line segments.

2. The method of claim 1 , wherein receiving sensor data representing aspects of the environment comprises:
receiving depth images representing aspects of the environment from a depth sensor.

3. The method of claim 1 , wherein identifying the set of edge points in the sensor data comprises:
determining covariance of data points in a cluster of data points in the sensor data; and
identifying data points in the cluster of data points that correspond to edge points based on the covariance of the data points.

4. The method of claim 1 , wherein determining the set of line segments from the set of edge points comprises:
selecting a first edge point and a second edge point from the detected edge points;
estimating a line segment model that includes at least the first edge point and the second edge point;
modifying the line segment model based on an additional edge point located proximate to the line segment model; and
based on the modified line segment model, determining a line segment that includes the first edge point, the second edge point, and the additional edge point.

5. The method of claim 1 , further comprising:
causing a robotic device to insert tines into pockets of the pallet; and
responsive to causing the robotic device to insert tines into pockets of the pallet, causing the robotic device to lift the pallet.

6. The method of claim 1 , further comprising:
responsive to receiving sensor data representing aspects of the environment, performing a filter process to remove data points of the sensor data that represent surfaces at amplitudes outside a range of amplitudes associated with a pallet positioned on a ground surface.

7. The method of claim 1 , wherein filtering the set of line segments to exclude line segments that have a length outside a height range and a width range associated with dimensions of a pallet template comprises:
determining an orientation of a line segment of the set of line segments;
based on the orientation of the line segment, determining whether a length of the line segment is outside either the height range or the width range associated with the dimensions of the pallet template; and
based on determining that the length of the line segment is outside either the height range or the width range, modifying the set of line segments to exclude the line segment.

8. The method of claim 1 , wherein identifying, from the filtered set of line segments, the subset of line segments that align with the pallet template comprises:
determining the subset of line segments that align with the dimensions of the pallet template using a registration process.

9. The method of claim 1 , further comprising:
determining a confidence level associated with an alignment of the subset of line segments with the dimensions of the pallet template; and
based on the confidence level, determining a control strategy for a robotic device using the pose of the pallet.

10. The method of claim 1 , wherein determining the pose of the pallet in the environment based on the identified subset of line segments comprises:
determining a distance and a relative position of the pallet from the sensor; and
determining an orientation of the pallet relative to the sensor.

11. A robotic device, comprising:
a sensor coupled to the robotic device; and
a control system configured to;
receive sensor data representing aspects of an environment, wherein the sensor data includes depth measurements of the environment from a three-dimensional (3D) camera;
identify a set of edge points in the sensor data, wherein identifying the set of edge points in the sensor data involves: (i) determining a cluster of local neighborhood data points in an depth image from the 3D camera, (ii) determining depth measurements of data points in the cluster of local neighborhood data points, and (iii) identifying data points of the cluster of local neighborhood data points that correspond to edge points based on the determined depth measurements;
determine a set of line segments from the set of edge points, wherein each line segment is fit to a subset of the set of edge points;
filter the set of line segments to exclude line segments that have a length outside a height range and a width range associated with dimensions of a pallet template;
identify, from the filtered set of line segments, a subset of line segments that align with the pallet template; and
determine a pose of a pallet in the environment based on the identified subset of line segments.

12. The system of claim 11 , wherein the robotic device is an autonomous robotic forklift.

13. The system of claim 11 , wherein the sensor data includes a color image from a set of stereo cameras, and
wherein the control system is configured to convert the color image into a grayscale image to identify the set of edge points in the sensor data.

14. The system of claim 11 , wherein the sensor data includes a depth image from a 3D camera, and
wherein the control system is further configured to:
determine a cluster of local neighborhood data points in the depth image;
determine depth measurements corresponding to data points in the cluster of local neighborhood data points; and
identify data points of the cluster of local neighborhood data points that correspond to edge points based on the determined depth measurements.

15. A non-transitory computer readable medium having stored therein program instructions executable by a computing system to cause the computing system to perform operations, the operations comprising:
receiving sensor data representing aspects of an environment;
identifying a set of edge points in the sensor data, wherein identifying the set of edge points involves: (i) determining a first local surface normal relative to a surface represented in a cluster of data points in the sensor data and a second local surface normal relative to the surface represented in the cluster of data points, and (ii) identifying data points in the cluster of data points that correspond to edge points based on the first local surface normal and the second local surface normal;
determining a set of line segments from the set of edge points, wherein each line segment is fit to a subset of the set of edge points;
filtering the set of line segments to exclude line segments that have a length outside a height range and a width range associated with dimensions of a pallet template;
identifying, from the filtered set of

Assignees

• X Dev Llc

Inventors

• Holz Dirk

Classifications

• G06T2207/10028

  Range image; Depth image; 3D point clouds · CPC title

• G06T7/13

  Edge detection · CPC title

• G06T7/75

  involving models · CPC title

• B66F9/063

  Automatically guided · CPC title

• B66F9/00

  Devices for lifting or lowering bulky or heavy goods for loading or unloading purposes (mobile or transportable lifts in, or associated with, buildings and specially adapted to be shifted from one part of a building or other structure to another part or to another building or structure B66B9/16; cranes B66C) · CPC title