Robotic system with automated package registration mechanism and methods of operating the same

Patent metadata

Field	Value
Publication number	US-11288810-B2
Application number	US-202016888376-A
Country	US
Kind code	B2
Filing date	May 29, 2020
Priority date	Oct 30, 2018
Publication date	Mar 29, 2022
Grant date	Mar 29, 2022

Abstract

A method for operating a robotic system includes obtaining and processing first data representative of an object at a start location. An event may be detected while implementing an operation based on the image data. A gripper height that corresponds to the event may be determined. Accordingly, the method may include calculating an object height that represents a height estimate of the object.

First claim

We claim:

1. A method for operating a robotic system, the method comprising:
receiving an image data representative of an object located at a start location;
implementing an operation based on the image data, wherein the operation is for transferring the object to a task location;
detecting an event while implementing the operation, wherein the event represents an entry of the object relative to a laterally-oriented crossing reference located at a crossing reference height;
determining a gripper height corresponding to the event, wherein the gripper height represents a vertical location of an end-effector; and
calculating an object height representing a height estimate of the object based on the event, the gripper height, and/or the predetermined reference height.

2. The method of claim 1 , wherein the event represents an output from a crossing sensor configured to detect optical signals transmitted along a line or a plane corresponding to the laterally-oriented crossing reference.

3. The method of claim 1 , further comprising obtaining profile data based on the event, wherein the profile data represents an output from a laterally-facing sensor configured to obtain information regarding a vertical edge, a vertical surface, or a combination thereof of the object.

4. The method of claim 3 , wherein the profile data includes a two-dimensional (2D) and/or a three-dimensional (3D) imaging data representative of one or more surfaces orthogonal to the surface represented in the image data of the object located at the start location.

5. The method of claim 1 , wherein:
the event represents the entry of an unrecognized object relative to the laterally-oriented crossing reference located above and overlapping the task location; and
calculating the object height includes calculating the object height based on the gripper height at a time when the event occurred.

6. The method of claim 5 , wherein calculating the object height includes:
identifying a time stamp corresponding to the entry of the unrecognized object relative to the laterally-oriented crossing reference;
determining the gripper height includes determining the gripper height at a time matching the time stamp; and
calculating the object height includes calculating the object height based on a difference between the gripper height and the predetermined reference height.

7. The method of claim 5 , wherein the event corresponds to a trigger for operating the end-effector to release the unrecognized object.

8. The method of claim 5 , wherein the event represents the entry of the unrecognized object relative to the laterally-oriented crossing reference located above a release trigger height for placing the object at the task location.

9. The method of claim 1 , further comprising:
comparing the image data to registration data representative of known objects for determining that the object is an unrecognized object; and
wherein:
the event is detected based on the determination that the object is unrecognizable, and
the event represents an arming event based on the entry of the unrecognized object relative to the laterally-oriented crossing reference located above and overlapping the start location, wherein the arming event is for detecting a subsequent exit event used to calculate the object height; and
determining the gripper height corresponding to the event includes determining the gripper height at a time of the exit event following the arming event; and
calculating the object height of the object based on a difference between the gripper height at the exit event and the predetermined reference height.

10. The method of claim 9 , further comprising:
tracking the gripper height during the operations for lifting the unrecognized object from the start location;
wherein:
detecting the event includes detecting the event when the gripper height is at or within a threshold distance from the predetermined reference height.

11. The method of claim 9 , wherein detecting the event includes detecting the event after operations for gripping the unrecognized object.

12. The method of claim 11 , wherein detecting the arming event is for activating a sensor corresponding to the laterally-oriented crossing reference.

13. The method of claim 11 , further comprising:
tracking information corresponding to an end-effector location contemporaneously with the operations that transfer the unrecognized object; and
wherein:
detecting the event includes detecting the event after an initial state and ignoring one or more sensor outputs during the initial state, wherein the initial state corresponds to when the gripper height is below the reference height and/or when horizontal coordinate values of the end-effector location are outside of a boundary associated with the start location, wherein the sensor outputs correspond to crossing events relative to the laterally-oriented crossing reference.

14. The method of claim 13 , further comprising:
identifying an external state during the operation, wherein the external state represents the end-effector location being outside of the boundary;
identifying an approach state during the operation, wherein the approach state represents the end-effector location being within the boundary and below the reference height; and
wherein:
ignoring the one or more sensor outputs during the initial state includes ignoring one or more sensor outputs during the external state and/or the approach state.

15. The method of claim 13 , wherein:
the end-effector location corresponds to a set of coordinate values representative of a location in three-dimensional space; and
the gripper height corresponds to a vertical coordinate value for the end-effector location.

16. A robotic system comprising:
at least one processor; and
at least one memory device connected to the at least one processor and having stored thereon instructions executable by the processor for:
obtaining image data representative of an object at a start location;
implementing a motion plan according to the image data, wherein the motion plan is transferring the object to a task location;
detecting an event that represents an entry of the object relative to a crossing reference during implementation of the motion plan;
determining a gripper height corresponding to the event, wherein the gripper height represents a vertical location of an end-effector; and
calculating an object height representing a height estimate of the object based on the event, the gripper height, and/or the predetermined reference height.

17. A tangible, non-transient computer-readable medium having processor instructions stored thereon that, when executed by a robotic system via one or more processors thereof, cause the robotic system to perform a method, the method comprising:
obtaining image data representative of an object located at a start location;
implementing an operation based on the image data, wherein the operation is for transferring the object to a task location;
detecting an event while implementing the operation, wherein the event represents an entry of the object relative to a laterally-oriented crossing reference located at a crossing reference height;
determining a gripper height corresponding to the event, wherein the gripper height represents a vertical location of an end-effector; and
calculating a height representing a height estimate of the object based on the event, the gripper height corresponding to the event, and/or the predetermined reference height.

18. The tangible, non-transient computer-rea

Assignees

• Mujin Inc

Inventors

• Diankov Rosen
• Liu Huan
• Ye Xutao
• Rodrigues Jose Jeronimo Moreira
• Kanemoto Yoshiki
• Yu Jinze
• Islam Russell

Classifications

• G06V10/44

  Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components · CPC title

• G06V10/255

  Detecting or recognising potential candidate objects based on visual cues, e.g. shapes · CPC title

• G06T7/13 Primary

  Edge detection · CPC title

• G06Q10/087 Primary

  Inventory or stock management, e.g. order filling, procurement or balancing against orders · CPC title

• G06V10/25

  Determination of region of interest [ROI] or a volume of interest [VOI] · CPC title