Robotic system with automated package registration mechanism and minimum viable region detection

Patent metadata

Field	Value
Publication number	US-11636605-B2
Application number	US-202117344720-A
Country	US
Kind code	B2
Filing date	Jun 10, 2021
Priority date	Oct 30, 2018
Publication date	Apr 25, 2023
Grant date	Apr 25, 2023

Abstract

The present disclosure relates to methods and systems for generating a verified minimum viable range (MVR) of an object. An exposed outer corner and exposed edges of an object may be identified by processing one or more image data. An initial MVR may be generated by identifying opposing parallel edges opposing the exposed edges. The initial MVR may be adjusted, and the adjusted result may be tested to generate a verified MVR.

First claim

What is claimed is:

1. A method comprising:
receiving one or more image data representative of a target stack including one or more unregistered objects;
identifying an exposed outer corner associated thereto based on the one or more image data, wherein the exposed outer corner corresponds to exposed edges that represent an unregistered object;
identifying complementing edges associated with the exposed edges based on identifying one or more other edges that intersect the exposed edges;
generating an initial minimum viable region (MVR) based on the exposed outer corner, the exposed edges, and the complementing edges; and
generating a verified MVR based on adjusting the initial MVR and testing a corresponding result, wherein the verified MVR represents an area between estimated peripheral edges of one of the unregistered objects.

2. The method of claim 1 , wherein:
the exposed edges include a first exposed edge and a second exposed edge that define the exposed outer corner;
the complementing edges are an initial set of complementing edges that include a first initial complementing edge and a second initial complementing edge, wherein the first initial complementing edge and the second initial complementing edge are determined to be parallel to the first exposed edge and the second exposed edge, respectively; and
generating the verified MVR includes:
identifying a further complementing edge that intersects the first exposed edge at a location past the second initial complementing edge relative to the exposed outer corner, wherein the further complementing edge is determined to be parallel to the second initial complementing edge and the second exposed edge;
identifying an additional plausible MVR region including an area between the complementing edges and the further complementing edge; and
testing a candidate MVR that combines the additional plausible MVR region and the initial MVR.

3. The method of claim 2 , wherein testing includes verifying whether one or more dimensions of the candidate MVR satisfy a maximum and/or a minimum dimension threshold for MVRs.

4. The method of claim 2 , wherein testing includes verifying whether depth measurements, colors, brightness, or a combination thereof for portions of the one or more imaging data corresponding to the additional plausible MFR region and the initial MVR are within a threshold continuity range from each other.

5. The method of claim 2 , wherein the verified MVR includes the initial MVR without the additional plausible MFR region when the candidate MVR fails to satisfy one or more predetermined thresholds.

6. The method of claim 1 , wherein:
the exposed edges include a first exposed edge and a second exposed edge that define the exposed outer corner;
the complementing edges are an initial set of complementing edges that include a first initial complementing edge and a second initial complementing edge, wherein the first initial complementing edge and the second initial complementing edge are determined to be parallel to the first exposed edge and the second exposed edge, respectively; and
generating the verified MVR includes:
identifying a further complementing edge between the second initial complementing edge and the exposed outer corner, wherein the further complementing edge is determined to be parallel to the second initial complementing edge and the second exposed edge;
identifying a reduced candidate MVR as an area within the initial MVR that is bounded by the further complementing edge instead of the second initial complementing edge; and
testing the reduced candidate MVR.

7. The method of claim 6 , wherein generating the verified MVR includes setting the reduced candidate MVR as the verified MVR when one or more dimensions of the candidate MVR satisfy a maximum and/or a minimum dimension threshold for MVRs.

8. The method of claim 6 , wherein identifying the further complementing edge includes:
identifying a second corner associated with the initial set of complementing edges;
searching for an intersection with the further complementing edge while moving away from the second corner along the first initial complementing edge; and
testing an orientation of the further complementing edge based on verifying (1) that an angle between the further complementing edge and the first initial complementing edge is within a threshold range representative of a right angle and/or (2) that a set of distances between multiple sets of corresponding points along the further complementing edge and the second exposed edge are within a threshold range from each other.

9. The method of claim 6 , wherein:
the one or more image data include (1) a 3D imaging output including depth measures for locations along a horizontal plane and (2) a 2D imaging output visually representing one or more of the unregistered objects along the horizontal plane;
identifying the exposed edges includes:
identifying each of the separated layers as a set of adjacent locations along the horizontal plane of the 3D imaging output that have the depth measures within a threshold range from each other, and
identifying peripheral boundaries of the separated layers as 3D edges;

identifying the exposed outer corner includes selecting an intersection formed by a set of the 3D edges, wherein the set of 3D edges are the exposed edges; and
identifying the further complementing edge includes identifying an incomplete edge and/or a detected line based on analyzing the 3D imaging output, the 2D imaging output, or a combination thereof.

10. The method of claim 9 wherein generating the verified MVR includes:
identifying a 3D edge based on analyzing the 3D imaging output, wherein the 3D edge (1) is closer towards the exposed outer corner along the first exposed edge than the second initial complementing edge and (2) is parallel to the second initial complementing edge and the second exposed edge;
adjusting the initial MVR to additionally include an area between the second initial complementing edge and the 3D edge;
wherein:
the further complementing edge is a result of analyzing portions of the 2D imaging output corresponding to the adjusted initial MVR; and
the reduced candidate MVR includes the area within the adjusted initial MVR.

11. A robotic system, comprising:
at least one processor; and
at least one memory device connected to the at least one processor and having stored thereon instructions, executable by the processor, for:
receiving one or more image data representative of one or more unregistered objects;
identifying an exposed outer corner based on the one or more image data, wherein the exposed outer corner represents an unregistered object;
generating an initial minimum viable region (MVR) based on the exposed outer corner; and
generating a verified MVR based on adjusting the initial MVR and testing a corresponding result.

12. The robotic system of claim 11 , wherein the at least one memory device includes instructions for:
receiving the one or more image data including 3D depth measures;
identifying the exposed outer corner based on:
identifying one or more surfaces based on the 3D depth measures, wherein each of the surfaces include a set of adjacent locations having the 3D depth measures within a threshold continuity range,
identifying 3D edges based on the identified surfaces, wherein the 3D edges correspond to lateral boundaries of the surfaces,
identifying the exposed outer corner as a junction between a first 3D edge and a second 3D edge bounding one of the surfaces, wherein the pair of the 3D edges correspond to the exposed edges; and

generating the initial MVR based on:
identifying a first complem

Assignees

• Mujin Inc

Inventors

• Yu Jinze
• Moreira Rodrigues Jose Jeronimo
• Diankov Rosen Nikolaev
• Ye Xutao

Classifications

• G05B2219/40006

  Placing, palletize, un palletize, paper roll placing, box stacking · CPC title

• G06V2201/06

  Recognition of objects for industrial automation · CPC title

• G06T2207/30164

  Workpiece; Machine component · CPC title

• G06T2207/10021

  Stereoscopic video; Stereoscopic image sequence · CPC title

• B65G2203/041

  Camera · CPC title