Robotic system with automated package registration mechanism and methods of operating the same

Patent metadata

Field	Value
Publication number	US-10703584-B2
Application number	US-201916539790-A
Country	US
Kind code	B2
Filing date	Aug 13, 2019
Priority date	Oct 30, 2018
Publication date	Jul 7, 2020
Grant date	Jul 7, 2020

Abstract

A method for operating a robotic system includes obtaining and processing first data representative of an object at a start location; obtaining additional data as the object is transferred to a task location, wherein the additional data includes information regarding one or more edges and/or one or more surfaces separate from portions of the object captured in the first data; and creating registration data representative of the object based on the additional data.

First claim

We claim:

1. A method for operating a robotic system, the method comprising:
receiving an image data representative of a surface of an object located at a start location;
determining a minimum viable region based on the image data for gripping and lifting the object;
obtaining data representing the object after implementation of an initial lift based on the minimum viable region to vertically displace the object;
implementing operations to transfer the object to a task location;
detecting an event and a corresponding time stamp while implementing the operation, wherein the event represents (1) an entry or an exit of the object relative to a laterally-oriented crossing reference located at a crossing reference height and (2) an output from a crossing sensor configured to detect optical signals transmitted along a line or a plane corresponding to the laterally oriented crossing reference;
determining a gripper height corresponding to the event, wherein the gripper height represents a vertical location of an end-effector;
calculating an object height of the object based on the event, the gripper height, and/or the predetermined reference height; and
creating new registration data representative of a new record for the object, wherein the new registration data includes the object height.

2. The method of claim 1 , further comprising obtaining profile data based on the event, wherein the profile data represents an output from a laterally-facing sensor configured to obtain information regarding a vertical edge, a vertical surface, or a combination thereof of the object.

3. The method of claim 2 , wherein the profile data includes a two-dimensional (2D) and/or a three-dimensional (3D) imaging data representative of one or more surfaces orthogonal to the surface represented in the image data of the object located at the start location.

4. The method of claim 1 , further comprising:
determining that the object is an unrecognized object based on comparing the image data to registration data representative of known objects; and
wherein:
the event is detected based on determining that the object is unrecognizable, and
the event represents the entry of the unrecognized object relative to the laterally-oriented crossing reference located above and overlapping the task location.

5. The method of claim 4 , wherein the event corresponds to a trigger for operating the end-effector to release the unrecognized object.

6. The method of claim 4 , wherein the event represents the entry or the exit of the unrecognized object relative to the laterally-oriented crossing reference located above a release trigger height.

7. The method of claim 1 , wherein:
determining that the object is an unrecognized object based on comparing the image data to registration data representative of known objects; and
wherein:
the event is detected based on determining that the object is unrecognizable, and
the event represents the exit of the unrecognized object relative to the laterally-oriented crossing reference located above and overlapping the start location.

8. The method of claim 7 , wherein detecting the event includes detecting the event contemporaneously corresponding to the operations for lifting the unrecognized object from the start location.

9. The method of claim 7 , wherein detecting the event includes detecting the event after operations for gripping the unrecognized object.

10. The method of claim 9 , wherein detecting the event includes activating a sensor corresponding to the laterally-oriented crossing reference.

11. The method of claim 9 , further comprising:
tracking an end-effector location contemporaneously with the operations that transfer the unrecognized object, wherein the end-effector location corresponds to a set of coordinate values; and
wherein:
the gripper height corresponds to a vertical coordinate value for the end-effector location; and
detecting the event includes ignoring an initial state when the gripper height is below the reference height and/or when horizontal coordinate values of the end-effector location are outside of a boundary associated with the start location, wherein the initial state represents one or more outputs from a sensor corresponding to the laterally-oriented crossing reference.

12. The method of claim 11 , further comprising:
identifying an external state during the operation, wherein the external state represents the end-effector location being outside of the boundary;
identifying an approach state during the operation, wherein the approach state represents the end-effector location being within the boundary and below the reference height; and
wherein:
ignoring the initial state includes ignoring the initial state during the external state and/or the approach state.

13. A robotic system comprising:
at least one processor; and
at least one memory device connected to the at least one processor and having stored thereon instructions executable by the processor for:
receiving an image data representative of a surface of an object located at a start location;
determining a minimum viable region based on the image data for gripping and lifting the object;
obtaining data representative of the object after implementation of an initial lift based on the minimum viable region to vertically displace the object;
implementing operations to transfer the object to a task location;
detecting an event during implementation of the operation, wherein the event represents (1) an entry or an exit of the object relative to a laterally-oriented crossing reference located at a crossing reference height and (2) an output from a crossing sensor configured to detect optical signals transmitted along a line or a plane corresponding to the laterally oriented crossing reference;
determining a gripper height corresponding to the event, wherein the gripper height represents a vertical location of an end-effector;
calculating an object height of the object based on the event, the gripper height, and/or the predetermined reference height; and
creating new registration data representative of the object, wherein the new registration data includes the object height.

14. A tangible, non-transient computer-readable medium having processor instructions stored thereon that, when executed by a robotic system via one or more processors thereof, cause the robotic system to perform a method, the method comprising:
receiving an image data representative of a surface of an object located at a start location;
determining a minimum viable region based on the image data for gripping and lifting the unrecognized object;
obtaining data representing the object after implementation of an initial lift based on the minimum viable region to vertically displace the object;
implementing operations to transfer the object to a task location;
detecting an event and a corresponding time stamp while implementing the operation, wherein the event represents (1) an entry or an exit of the unrecognized object relative to a laterally-oriented crossing reference located at a crossing reference height and (2) an output from a crossing sensor configured to detect optical signals transmitted along a line or a plane corresponding to the laterally oriented crossing reference;
determining a gripper height corresponding to the event, wherein the gripper height represents a vertical location of an end-effector;
calculating a height of the object based on the event, the gripper height corresponding to the event, and/or the predetermined reference height; and
creating new registration data representative of a new record for the object, wherein

Assignees

• Mujin Inc

Inventors

• Diankov Rosen Nikolaev
• Liu Huan
• Ye Xutao
• Rodrigues Jose Jeronimo Moreira
• Kanemoto Yoshiki
• Yu Jinze
• Islam Russell

Classifications

• G06V10/44

  Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components · CPC title

• G06V10/255

  Detecting or recognising potential candidate objects based on visual cues, e.g. shapes · CPC title

• G06T7/13

  Edge detection · CPC title

• G06Q10/087 Primary

  Inventory or stock management, e.g. order filling, procurement or balancing against orders · CPC title

• G06V10/25

  Determination of region of interest [ROI] or a volume of interest [VOI] · CPC title