Robotic system with wall-based packing mechanism and methods of operating same

Patent metadata

Field	Value
Publication number	US-11020854-B2
Application number	US-202016905837-A
Country	US
Kind code	B2
Filing date	Jun 18, 2020
Priority date	Nov 5, 2019
Publication date	Jun 1, 2021
Grant date	Jun 1, 2021

Abstract

A system and method for operating a robotic system to place objects into containers that have support walls is disclosed. The robotic system may derive a packing plan for stacking objects on top of each other. The robotic system may derive placement locations for one or more objects overhanging one or more support objects below. The derived placement locations may be based on utilizing one or more of the support walls to secure the placed object.

First claim

We claim:

1. A method for operating a robotic system, the method comprising:
obtaining a first object model and a second object model, wherein the first object model and the second object model are discretized representations of physical dimensions, shapes, or a combination thereof for a first object and a second object, respectively, according to a discretization unit, the first object model and the second object model representing the first object and the second object, respectively, designated for placing in one or more containers at a task location;
accessing a container model representative of a container having a support wall that extends above a container base, wherein the container model is a discretized representation of the container according to the discretization unit;
deriving a packing plan based on overlaying the first object model and the second object model over the discretized container model, the packing plan including a first placement location for placing the first object and a second placement location for placing the second object, wherein:
the first placement location represents a location within the container directly adjacent to the support wall,
the second placement location represents a location above the first placement location and within the container where a portion of the second object laterally protrudes beyond a peripheral edge of the first object toward the support wall, and
the second placement location is for utilizing the support wall to support the second object placed above the first object; and

implementing the packing plan for placing the first object and the second object in the container using a robotic arm and an end-effector.

2. The method of claim 1 , wherein:
the first placement location is for placing the first object at a location separated from the support wall by an object-wall separation distance; and
the second placement location is for placing the second object overhanging the first object.

3. The method of claim 1 , wherein deriving the packing plan includes:
deriving a candidate position within the discretized container model for placing the second object;
estimating one or more attributes associated with the candidate position based on placing the first object model at the first placement location and the second object model at the candidate position; and
validating the candidate position as the second placement location when the estimated attributes satisfy a wall-support rule that represents one or more requirements for placing objects against and directly contacting vertically-oriented container portions.

4. The method of claim 3 , wherein:
estimating the one or more attributes includes calculating an effective support based on a portion of the second object model overlapping the first object model; and
validating the candidate position includes determining that the effective support satisfies an overlap requirement.

5. The method of claim 3 , wherein:
estimating the one or more attributes includes estimating an overhang measure for a portion of the second object model overhanging the first object model; and
validating the candidate position includes determining that the overhang measure satisfies an overhang requirement.

6. The method of claim 3 , wherein:
estimating the one or more attributes includes estimating a center-of-mass (CoM) location corresponding to the CoM of the second object; and
validating the candidate position includes determining that the CoM location satisfies a CoM offset requirement.

7. The method of claim 1 , wherein deriving the packing plan includes:
deriving a candidate position within the discretized container model for placing the second object separated from the support wall;
estimating a pivot location associated with the candidate position based on placing the first object model at the first placement location and the second object model at the candidate position, wherein the pivot location represents a peripheral edge or a highest portion of the first object;
deriving one or more shifted poses based on shifting the second object model toward the support wall and/or rotating the second object model about the pivot location, wherein the one or more shifted poses represent the second object shifting from the candidate location and directly contacting the support wall; and
validating the candidate position as the second placement location when the one or more shifted poses satisfy a tilt-support rule that represents one or more requirements for placing objects laterally separated from vertically-oriented container portions.

8. The method of claim 1 , wherein deriving the packing plan includes:
deriving a candidate position within the discretized container model for placing the second object overhanging an intermediate object that is overhanging the first object;
estimating one or more attributes associated with the candidate position based on placing the first object model at the first placement location and the second object model at the candidate position above a location for the intermediate object; and
validating the candidate position as the second placement location when the estimated attributes satisfy a multiple overhang rule that represents one or more requirements for placing multiple objects overhanging a bottom support object.

9. The method of claim 8 , wherein:
estimating the one or more attributes includes calculating an effective support based on a portion of the second object model overlapping the first object model; and
validating the candidate position includes determining that the effective support satisfies an overlap requirement.

10. The method of claim 8 , wherein:
estimating the one or more attributes includes:
deriving a combined object estimation that represents the second object and the intermediate object as one object; and
estimating the one or more attributes of the combined object estimation; and

validating the candidate position includes determining that the one or more attributes satisfy a wall-support rule or a tilt-support rule.

11. The method of claim 1 , further comprising:
determining object groupings for objects designated for placing within containers at the task location;
deriving a processing order for the object groupings; and
wherein:
deriving the packing plan includes deriving the packing plan based on the processing order and the object groupings.

12. The method of claim 1 , wherein the packing plan is derived dynamically during or immediately before a packing operation.

13. The method of claim 12 , further comprising:
obtaining image data representing the container at the task location;
detecting an unexpected feature based on analyzing the image data, the unexpected feature representing an actual physical trait of the support wall that deviates from an expected condition; and
wherein:
deriving the packing plan includes dynamically deriving the packing plan based on the unexpected feature, wherein the packing plan is for replacing a preceding plan for placing the first object and the second object in the container according to the expected condition.

14. The method of claim 1 , further comprising:
obtaining a gripper model representative of the end-effector;
identifying one or more potential obstacles for placing the second object, wherein the one or more potential obstacles include a vertically-oriented portion of the container;
wherein deriving the packing plan includes:
deriving a candidate position within the discretized container model for placing the second object; and
deriving an approach path based on overlaying a combination

Assignees

• Mujin Inc

Inventors

• Kanunikov Denys
• Diankov Rosen Nikolaev

Classifications

• G06T2207/10028

  Range image; Depth image; 3D point clouds · CPC title

• G06T2207/10024

  Color image · CPC title

• G06T2200/04

  involving 3D image data · CPC title

• G06T7/74

  involving reference images or patches · CPC title

• G06T7/62

  of area, perimeter, diameter or volume · CPC title