Method for maintaining inventory in a store

Patent metadata

Field	Value
Publication number	US-11544668-B2
Application number	US-202217836778-A
Country	US
Kind code	B2
Filing date	Jun 9, 2022
Priority date	May 19, 2016
Publication date	Jan 3, 2023
Grant date	Jan 3, 2023

Abstract

A method for maintaining inventory within a store includes: accessing an image (e.g., a color image, depth image) depicting an inventory structure; detecting a slot region of the image depicting a slot; identifying a product type assigned to the slot; accessing a product dimension of the product type; defining a target region within the slot in the image based on the product dimension; defining a product region within the slot in the image based on the product dimension and the target region; defining a back-of-shelf plane intersecting the target region of the image; detecting a surface within the product region; and, in response to the surface intersecting the back-of-shelf plane, identifying the slot as empty and generating a prompt to restock the slot with product units of the product type.

First claim

We claim:

1. A method comprising, at a first time:
dispatching a mobile robotic system to automatically navigate along an inventory structure and capture a set of depth images depicting the inventory structure;
accessing a first depth image, in the set of depth images, of the inventory structure, the first depth image captured by the mobile robotic system at a first time;
detecting a first slot region of the first depth image depicting a first slot;
identifying a first product type assigned to the first slot;
accessing a first product dimension of the first product type;
detecting a first shelf face in the first depth image;
defining a first target region, offset above the first shelf face by the first product dimension, in the first depth image;
defining a first product region, between the first shelf face and the first target region, in the first depth image;
defining a first back-of-shelf plane intersecting the first target region of the first depth image;
detecting a first depth of a first surface within the first product region; and
in response to the first depth of the first surface intersecting the first back-of-shelf plane:
identifying the first slot as empty; and
generating a prompt to restock the first slot with product units of the first product type.

2. The method of claim 1 :
further comprising:
detecting a second slot region of the first depth image, adjacent the first slot region, representing a second slot;
identifying a second product type assigned to the second slot;
accessing a second product dimension of the second product type;
defining a second target region, offset above the first shelf face by the second product dimension, in the first depth image; and
defining a second product region, between the first shelf face and the second target region, in the first depth image;

wherein defining the first back-of-shelf plane comprises defining the first back-of-shelf plane intersecting the first target region and the second target region of the first depth image; and
further comprising:
calculating an offset distance between a second surface within the second product region of the first depth image and the first back-of-shelf plane; and
in response to the offset distance between the second surface and the first back-of-shelf plane exceeding a threshold difference, identifying the second slot as occupied.

3. The method of claim 1 :
wherein detecting the first slot region of the first depth image depicting the first slot comprises detecting the first slot region of the first depth image depicting the first slot in a first inventory section of the inventory structure; and
further comprising
detecting a second slot region of the first depth image depicting a second slot in a second inventory section of the inventory structure;
identifying a second product type assigned to the second slot;
accessing a second product dimension of the second product type;
detecting a second shelf face in the first depth image;
defining a second target region, offset above the second shelf face by the second product dimension, in the first depth image;
defining a second product region, between the second shelf face and the second target region, in the first depth image;
defining a second back-of-shelf plane intersecting the second target region of the first depth image, separate from the first back-of-shelf plane;
detecting a second depth of a second surface within the second product region; and
in response to the second depth of the second surface intersecting the second back-of-shelf plane:
identifying the second slot as empty; and
generating a prompt to restock the second slot with product units of the second product type.

4. The method of claim 1 , further comprising:
detecting a first top edge of the first shelf face in the first depth image;
defining a first top-of-shelf plane intersecting the first top edge of the first shelf face;
detecting a second slot region of the first depth image, below the first slot region, representing a second slot;
identifying a second product type assigned to the second slot;
accessing a second product dimension of the second product type;
detecting a second shelf face below the first shelf face in the first depth image;
defining a second target region, offset above the second shelf face by the second product dimension, in the first depth image;
defining a second product region, between the second shelf face and the second target region, in the first depth image; and
in response to the first back-of-shelf plane in the second target region obstructed by the first shelf face in the first depth image:
extending the first back-of-shelf plane from the first slot region to intersect the second slot region;
detecting a second top edge of the second shelf face in the first depth image;
defining a second top of shelf plane intersecting the second top edge of the second shelf face;
detecting a second depth of a second surface, above the second top-of-shelf plane, within the second slot region;
calculating an offset distance between the second surface within the second slot region and the first back-of-shelf plane; and
in response to the offset distance between the second surface and the first back-of-shelf plane exceeding a threshold difference, identifying the second slot as occupied.

5. The method of claim 1 :
further comprising, detecting a second shelf face above the first shelf face; and
wherein defining the first target region comprises defining the first target region offset above the first shelf face by the first product dimension and extending to the second shelf face, in the first depth image.

6. The method of claim 1 :
further comprising:
accessing a first color image captured concurrently with the first depth image;
detecting a first slot tag associated with the first slot in the first color image; and
detecting a first product identifier in the first slot tag in the first color image; and

wherein identifying the first product type assigned to the first slot comprises identifying the first product type of the first slot based on the first product identifier.

7. The method of claim 1 :
further comprising:
detecting a second slot region of the first depth image, below the first slot region, representing a second slot;
identifying a second product type assigned to the second slot; and
accessing a second product dimension of the second product type, different from the first product dimension; and

wherein defining the first target region, offset above the first shelf face by the first product dimension, in the first depth image comprises defining the first target region, offset above the first shelf face by the greater of the first product dimension and the second product dimension.

8. The method of claim 1 , further comprising:
at the first time, further comprising recording the first back-of-shelf plane in a database; and
at a second time further comprising:
accessing a second depth image of the inventory structure, the second depth image captured by the mobile robotic system at a second time;
detecting a second slot region of the second depth image depicting the first slot;
detecting a second shelf face in the second depth image;
defining a second target region, offset above the second shelf face by the first product dimension, in the second depth image;
defining a second product region, between the second shelf face and the second target region, in the second depth image;
identifying the second target region as obstructed in the second depth image; and
in response to identifying the second target region as obstructed in the second depth image;
accessing the first back-of-shelf plane from the database;
proj

Assignees

• Simbe Robotics Inc

Inventors

• Bahirat Kanchan
• Reddy Shiva
• Bogolea Bradley
• Safi Jariullah

Classifications

• H04N25/615

  involving a transfer function modelling the optical system, e.g. optical transfer function [OTF], phase transfer function [PhTF] or modulation transfer function [MTF] · CPC title

• G06V20/52

  Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69) · CPC title

• G06V10/987

  with the intervention of an operator · CPC title

• G06F18/22

  Matching criteria, e.g. proximity measures · CPC title

• H04N23/698

  for achieving an enlarged field of view, e.g. panoramic image capture · CPC title