Visual perception and techniques for placing inventory into pods with a robotic workcell

Patent metadata

Field	Value
Publication number	US-12487587-B1
Application number	US-202217707579-A
Country	US
Kind code	B1
Filing date	Mar 29, 2022
Priority date	Mar 29, 2022
Publication date	Dec 2, 2025
Grant date	Dec 2, 2025

Abstract

Techniques and systems for performing a perception analysis for a robotic stowing operation are described. An example technique includes obtaining, via multiple sensors, multiple first images, wherein each first image is an image of a different container of an inventory holder within a robotic workcell. A first machine learning (ML) and image processing pipeline is performed with the first images to determine displacement locations for the containers of the inventory holder. A second ML and image processing pipeline is performed with the first images to determine content signatures for the containers. A plan is generated for stowing a first item into a first container, based at least in part on the plurality of content signatures and the plurality of displacement locations. A robotic apparatus is controlled to stow the first item into the first container, based on the plan.

First claim

What is claimed is:

1 . An inventory system comprising:
a plurality of sensors disposed within a robotic workcell; an inventory holder comprising a plurality of containers, wherein each of the plurality of containers comprises a plurality of retaining elements extending across an opening of the container; a robotic apparatus comprising a plurality of end effector tools; and a controller configured to perform an operation for stowing an item into one of the plurality of containers using the robotic apparatus, the operation comprising:
capturing, via the plurality of sensors, a first image of the inventory holder;
generating, from the first image, a plurality of second images, wherein each second image is an image of a different container of the plurality of containers;
determining a plurality of displacement locations for the plurality of containers, based on the plurality of second images, wherein each displacement location corresponds to a location within the respective container for placing a first end effector tool of the plurality of end effector tools to displace the plurality of retaining elements extending across the opening of the container;
determining a plurality of content signatures for the plurality of containers, based on the plurality of second images, wherein each content signature includes an indication of one or more regions within the container available for stowing an item into the container;
generating a match result comprising at least (i) an indication of a first container of the plurality of containers in which to stow a first item, (ii) a grasping strategy for grasping the first item, and (iii) a placement strategy for placing the first item into the one or more regions of the first container using a second end effector tool of the plurality of end effector tools;
controlling the first end effector tool to displace the plurality of retaining elements extending across the opening of the first container at the displacement location corresponding to the first container;
controlling the second end effector tool to grasp the first item using the grasping strategy; and
controlling the second end effector tool to place the first item that is grasped into the one of the one or more regions of the first container using the placement strategy.

2 . The inventory system of claim 1 , wherein generating the plurality of second images comprises:
generating a third image based on performing semantic segmentation on the first image, wherein the third image comprises an indication of the plurality of containers segmented from the inventory holder; and performing one or more image processing operations on the third image to generate the plurality of second images.

3 . The inventory system of claim 2 , wherein the one or more image processing operations comprise at least one of (i) a homography operation or (ii) an extraction operation.

4 . A computer-implemented method for performing a perception analysis for a robotic stowing operation, the computer-implemented method comprising:
obtaining, via a plurality of sensors, a plurality of first images, wherein each first image is an image of a different container of an inventory holder within a robotic workcell; performing a first machine learning and image processing pipeline with the plurality of first images to determine a plurality of displacement locations for a plurality of containers of the inventory holder; performing a second machine learning and image processing pipeline with the plurality of first images to determine a plurality of content signatures for the plurality of containers; generating a plan for stowing a first item into a first container of the plurality of containers, based at least in part on the plurality of content signatures and the plurality of displacement locations; and controlling a robotic apparatus to stow the first item into the first container, based on the plan.

5 . The computer-implemented method of claim 4 , wherein the first machine learning and image processing pipeline and the second machine learning and image processing pipeline are performed in parallel.

6 . The computer-implemented method of claim 4 , wherein each of the plurality of first images comprises an indication of one or more retaining elements disposed at an opening of the respective container.

7 . The computer-implemented method of claim 6 , wherein performing the first machine learning and image processing pipeline comprises, for each first image:
performing a semantic segmentation operation on the first image to generate a mask indicating regions within the first image belonging to the one or more retaining elements; and performing one or more image processing operations on the mask to identify one or more clusters of pixels within the regions belonging to the one or more retaining elements.

8 . The computer-implemented method of claim 7 , wherein performing the first machine learning and image processing pipeline further comprises, for each first image, selecting a location within the mask that satisfies a predetermined condition as a displacement location for the respective container.

9 . The computer-implemented method of claim 4 , wherein controlling the robotic apparatus comprises controlling movement of an end effector tool of the robotic apparatus to displace one or more retaining elements disposed at an opening of the first container at a displacement location corresponding to the first container.

10 . The computer-implemented method of claim 4 , wherein performing the second machine learning and image processing pipeline comprises, for each first image:
performing a semantic segmentation operation on the first image to generate a mask indicating regions within the first image belonging to unoccupied space; and performing one or more image processing operations on the mask to determine one or more regions within the respective container available for stowing an item, wherein a content signature for the respective container includes the one or more regions.

11 . The computer-implemented method of claim 10 , wherein the one or more regions are indicated as vertical slot regions within the first image.

12 . The computer-implemented method of claim 10 , wherein each of the one or more regions is associated with a placement strategy for placing an item into the region using an end effector tool of the robotic apparatus.

13 . The computer-implemented method of claim 12 , wherein the placement strategy comprises a bin sweep motion or a slot wedge motion.

14 . The computer-implemented method of claim 4 , wherein performing the second machine learning and image processing pipeline comprises, for each first image performing an instance segmentation operation on the first image to generate a mask indicating one or more regions within the first image belonging to one or more different instances of items, wherein a content signature for the respective container includes the one or more regions.

15 . The computer-implemented method of claim 4 , wherein generating the plan comprises generating a ranked list of container-to-item matches, wherein each container-to-item match in the ranked list is associated with a confidence score.

16 . The computer-implemented method of claim 15 , wherein generating the plan further comprises selecting, from the ranked list of container-to-item matches, a container-to-item match having a confidence score that satisfies a predetermined condition, wherein the selected container-to-item match indicates (i) the first container of the plurality of containers for stowi

Assignees

• Amazon Tech Inc

Inventors

• Owan Parker
• Karumanchi Sisir Babu
• Klinkner Kristina Lisa
• Wang Xin
• Liu Baoyuan
• Hooks Joshua Rosenberg
• Warrier Rahul Balakrishna
• Cortes Andres Camilo
• Dai Siyu
• Parness Aaron Joseph
• Schmidt James
• Stubbs Andrew

Classifications

• B25J9/1687

  Assembly, peg and hole, palletising, straight line, weaving pattern movement · CPC title

• B25J9/1612

  characterised by the hand, wrist, grip control · CPC title

• B25J9/1697

  Vision controlled systems · CPC title

• B25J9/1682

  Dual arm manipulator; Coordination of several manipulators · CPC title

• B25J9/1615

  characterised by special kind of manipulator, e.g. planar, scara, gantry, cantilever, space, closed chain, passive/active joints and tendon driven manipulators · CPC title