Robotic system with automated package scan and registration mechanism and methods of operating the same

Patent metadata

Field	Value
Publication number	US-11501445-B2
Application number	US-201916667857-A
Country	US
Kind code	B2
Filing date	Oct 29, 2019
Priority date	Oct 30, 2018
Publication date	Nov 15, 2022
Grant date	Nov 15, 2022

Abstract

A system and method for operating a robotic system to scan and register unrecognized objects is disclosed. The robotic system may use an image data representative of an unrecognized object located at a start location to implement operations for transferring the unrecognized object from the start location. While implementing the operations, the robotic system may obtain additional data, including scanning results of one or more portions of the unrecognized object not included in the image data. The robotic system may use the additional data to register the unrecognized object.

First claim

We claim:

1. A method for operating a robotic system, the method comprising:
receiving an image data representative of a top surface of an unrecognized object located at a start location;
identifying a first set of peripheral edges of the top surface represented in the image data;
implementing operations for transferring the unrecognized object to a task location based at least partially on the first set of peripheral edges;
detecting an event while implementing the operations for transferring the unrecognized object, wherein the event represents an entry or an exit of the unrecognized object relative to a laterally-oriented crossing reference located at a crossing reference height;
calculating an object height of the unrecognized object based on the event;
deriving a location of a vertically-oriented surface or a portion thereof of the unrecognized object based on the object height and the first set of peripheral edges;
deriving a scanning position based on the derived location of the vertically-oriented surface, wherein the scanning position represents a pose for presenting the vertically-oriented surface to a scanning sensor;
implementing operations for transferring the unrecognized object to expose the vertically-oriented surface to a scanning sensor based on the scanning position;
generating a scanning result representative of an output from the scanning sensor scanning the vertically-oriented surface or the portion thereof; and
creating new registration data representative of a new record for the unrecognized object, wherein the new registration data includes the scanning result.

2. The method of claim 1 , further comprising:
generating one or more commands for operating the scanning sensor based on implementing the operations to transfer the unrecognized object to expose the vertically-oriented surface to the scanning sensor; and
wherein:
the scanning result is a value represented by a bar code and/or a Quick Response (QR) code on the vertically-oriented surface.

3. The method of claim 1 , further comprising:
deriving a scanning maneuver based on the scanning position, wherein the scanning maneuver is for transferring the unrecognized object to present multiple portions thereof to the scanning sensor;
after implementing operations to transfer the vertically-oriented surface, implementing operations according to the scanning maneuver for presenting multiple portions of the vertically-oriented surface and/or multiple surfaces of the unrecognized object to the scanning sensor; and
generating one or more commands for operating the scanning sensor based on implementing the operations according to the scanning maneuver to scan the multiple portions of the vertically-oriented surface and/or the multiple surfaces.

4. The method of claim 3 , wherein the scanning maneuver corresponds to rotating the unrecognized object about a vertical axis.

5. The method of claim 3 , further comprising:
calculating a dimension of the vertically-oriented surface based on the object height and/or the first set of peripheral edges;
deriving a second scanning position based on the derived location of the vertically-oriented surface when the dimension exceeds a threshold, wherein the scanning position and the second scanning position are for presenting different end portions or corners of the vertically-oriented surface to the scanning sensor;
wherein:
the scanning maneuver is for transferring the unrecognized object from the scanning position to the second scanning position; and
the one or more commands for operating the scanning sensor represents keeping the scanning sensor active while implementing operations according to the scanning maneuver.

6. A robotic system comprising:
at least one processor; and
at least one memory device connected to the at least one processor and having stored thereon instructions executable by the processor for:
receiving an image data representative of an unrecognized object located at a start location;
implementing a task based on the image data, wherein the task includes operations for transferring the unrecognized object to a task location;
while implementing the operations for transferring the unrecognized object, implement operations for scanning at least one portion of the unrecognized object that was not represented in the image data;
generating a scanning result based on implementing the operations for scanning the at least one portion of the unrecognized object; and
creating new registration data representative of a new record for the unrecognized object, wherein the new registration data includes the scanning result.

7. The robotic system of claim 6 , wherein the at least one memory device includes instructions for:
obtaining additional data corresponding to the unrecognized object while implementing the task;
calculating an object height of the unrecognized object based on the additional data;
deriving a scanning position based on the object height, wherein the scanning position represents a pose for presenting to a scanning sensor the at least one portion of the unrecognized object that was not represented in the image data;
implementing operations for scanning at least one portion of the unrecognized object based on:
a command and/or a setting for positioning the unrecognized object according to the scanning position; and
an activation command for operating the scanning sensor while and/or after positioning the unrecognized object according to the scanning position.

8. The robotic system of claim 7 , wherein:
the image data represents a top surface of the unrecognized object;
the at least one memory device includes instructions for:
identifying locations of one or more peripheral edges of the top surface represented in the image data;
deriving a target location of the unrecognized object based on the one or more peripheral edges and the object height, wherein the target location represents a position of the at least one portion of the unrecognized object that was not represented in the image data;
determining a scanning zone of the scanning sensor, wherein the scanning zone represents a space detectable by the scanning sensor; and
deriving the scanning position based on the target location and the scanning zone.

9. The robotic system of claim 7 , wherein the at least one portion of the unrecognized object that was not represented in the image data includes a vertically-oriented surface or a peripheral portion thereof of the unrecognized object.

10. The robotic system of claim 9 , wherein:
the scanning position corresponds to a first corner of the vertically-oriented surface; and
the at least one memory device includes instructions for implementing operations for scanning based on:
a second scanning position derived based on the object height, wherein the second scanning position represents a pose for presenting to the scanning sensor a second corner of the vertically-oriented surface; and
a scanning maneuver derived according to the scanning position and the second scanning position for vertically and/or horizontally moving the vertically-oriented surface.

11. The robotic system of claim 10 , wherein the at least one memory device includes instructions for generating the scanning result including an identifier location based on:
receiving an identifier value resulting from scanning an identifier accessible on the vertically-oriented surface;
determining a gripper location at a time of receiving the identifier value;
deriving an identifier location based on the gripper location, wherein the identifier location represents a position of the identifier on the vertically-oriented surface; and
generating the s

Assignees

• Mujin Inc

Inventors

• Diankov Rosen Nikolaev
• Liu Huan
• Ye Xutao
• Rodrigues Jose Jeronimo Moreira
• Kanemoto Yoshiki
• Yu Jinze
• Islam Russell

Classifications

• G06V10/44

  Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components · CPC title

• G06V10/255

  Detecting or recognising potential candidate objects based on visual cues, e.g. shapes · CPC title

• G06T7/13 Primary

  Edge detection · CPC title

• G06Q10/087 Primary

  Inventory or stock management, e.g. order filling, procurement or balancing against orders · CPC title

• G06V10/25

  Determination of region of interest [ROI] or a volume of interest [VOI] · CPC title