Method for classifying product organization in an inventory structure

Patent metadata

Field	Value
Publication number	US-12536495-B2
Application number	US-202318101539-A
Country	US
Kind code	B2
Filing date	Jan 25, 2023
Priority date	May 19, 2016
Publication date	Jan 27, 2026
Grant date	Jan 27, 2026

Abstract

A method includes: locating a first slot boundary around a first product unit, in an image of an inventory structure in a store, identifies as a first product type based on features detected in the image; extracting a first product identifier, from a first slot tag, detected in the image; based on correspondence between the first product identifier and product type, associating the first slot boundary with the first slot tag and extracting a first relative tag-boundary position of the first slot boundary and slot tag from the image; extracting a second product identifier from a second slot tag detected in the image; predicting a second slot boundary, assigned to a second product type corresponding to the second product identifier, in the image based on a second position of the second tag in the image and the first relative tag-boundary position.

First claim

We claim:

1 . A method comprising:
by a mobile robotic system:
autonomously navigating throughout a store during a scan cycle; and
via an optical sensor integrated in the mobile robotic system, capturing images of inventory structures in the store during the scan cycle; and
by a computer system:
accessing a first image depicting an inventory structure in the store at a first time during the scan cycle, the first image captured by the mobile robotic system;
detecting a first product unit in the first image;
extracting a first set of features, depicting the first product unit, from the first image;
identifying the first product unit as a first product type based on the first set of features;
locating a first slot boundary in the first image, the first slot boundary containing the first product unit and defining a first rectilinear slot boundary;
detecting a first slot tag in the first image;
extracting a first product identifier, from the first slot tag, depicted in the first image;
in response to the first product identifier corresponding to the first product type:
associating the first slot boundary with the first slot tag;
extracting a first relative tag-boundary position of the first slot boundary and the first slot tag from the first image, the first relative tag-boundary position defining a shelf tag location adjacent and below a first lower-left corner of the first rectilinear slot boundary based on positions of the first rectilinear slot boundary and the first slot tag in the first image; and
identifying the first slot boundary as in-stock at the first time;
detecting a second slot tag in the first image; extracting a second product identifier, from the second slot tag, depicted in the first image; defining a second slot boundary assigned to a second product type corresponding to the second product identifier, the second slot boundary defining a second rectilinear slot boundary analogous to the first rectilinear slot boundary; locating a second lower-left corner of the second slot boundary adjacent and above the second slot tag in the first image; scanning the first image for a second product unit corresponding to the second product identifier; and in response to absence of the second product unit corresponding to the second product identifier within the second slot boundary in the first image, identifying the second slot boundary as out-of-stock at the first time.

2 . The method of claim 1 :
further comprising, by the computer system, deploying the mobile robotic system to autonomously navigate throughout the store during the scan cycle; and wherein accessing the first image comprises:
accessing a sequence of photographic images captured by the mobile robotic system during the scan cycle while traversing an aisle facing the inventory structure; and
compiling the sequence of photographic images into the first image defining a composite photographic image depicting a set of shelving segments spanning the inventory structure.

3 . The method of claim 1 , wherein predicting the second slot boundary in the first image comprises defining the second slot boundary, adjacent and disjoint from the first slot boundary, in the first image based on the second position of the second slot tag in the first image and the first relative tag-boundary position.

4 . The method of claim 1 :
wherein extracting the first set of features from the first image comprises extracting the first set of features comprising color values and text strings from a first region of the first image depicting the first product unit; and wherein identifying the first product unit as the first product type comprises:
retrieving a set of visual feature templates representing a set of product types in a product category associated with the inventory structure;
scanning the set of visual feature templates for features matching color values and text strings in the first set of features; and
identifying the first product unit as the first product type in response to the first set of features approximating a first visual feature template corresponding to the product type.

5 . The method of claim 1 , further comprising:
storing a first location of the first slot tag in a realogram of the store; storing the first slot boundary, associated with the first slot tag, in the realogram; storing a second location of the second slot tag in the realogram; and temporarily storing the second slot boundary, associated with the second slot tag, in the realogram.

6 . The method of claim 5 , further comprising:
accessing a second image depicting the inventory structure at a second time succeeding the first time; detecting a third product unit in the second image; extracting a third set of features, depicting the third product unit, from the second image; identifying the third product unit as the second product type based on the third set of features; locating a revised second slot boundary, containing the third product unit, in the second image; detecting the second slot tag in the second image; extracting the second product identifier, from the second slot tag, depicted in the second image; in response to the second product identifier corresponding to the second product type, associating the revised second slot boundary with the second slot tag; and in response to the second slot boundary differing from the revised second slot boundary:
replacing the second slot boundary with the revised second slot boundary in the realogram.

7 . The method of claim 6 , wherein accessing the second image comprises accessing the second image comprising a photographic image captured by a fixed camera, arranged within the store and facing the inventory structure, at the second time.

8 . The method of claim 6 , wherein accessing the second image comprises accessing the second image captured by the mobile robotic system during a second scan cycle succeeding the first scan cycle.

9 . The method of claim 1 :
further comprising:
detecting a third slot tag in the first image; and
extracting a third product identifier, from the third slot tag, depicted in the first image; and
wherein defining the second slot boundary comprises:
characterizing a first distance between the first slot tag and the second slot tag;
characterizing a second distance between the third slot tag and the second slot tag; and
in response to the second distance exceeding the first distance:
predicting correspondence of slot format between the first slot tag and the second slot tag; and
predicting the second slot boundary in the first image based on:
the second position of the second tag in the first image; and
the first relative tag-boundary position.

10 . The method of claim 9 :
further comprising:
detecting a fourth slot tag in the first image;
extracting a fourth product identifier, from the fourth slot tag, depicted in the first image;
characterizing a first height of the first slot tag in the first image;
characterizing a second height of the second slot tag in the first image;
characterizing a third height of the third slot tag in the first image; and
characterizing a fourth height of the fourth slot tag in the first image; and
wherein predicting correspondence of slot format between the first slot tag and the second slot tag comprises:
associating the first slot tag, the second slot tag, and the third slot tag with a first slot row in the inventory structure based on the first height, the second height, and the third height;
associating the fourth slot tag with a second slot row in the inventory structure based on the fourth height; and
predicting correspondence of slot format between the firs

Assignees

• Simbe Robotics Inc

Inventors

• Bogolea Brad
• Safi Jariullah
• Bahirat Kanchan
• Reddy Shiva

Classifications

• G05D1/0246

  using a video camera in combination with image processing means · CPC title

• H04N25/615

  involving a transfer function modelling the optical system, e.g. optical transfer function [OTF], phase transfer function [PhTF] or modulation transfer function [MTF] · CPC title

• H04N23/80

  Camera processing pipelines; Components thereof · CPC title

• H04N23/698

  for achieving an enlarged field of view, e.g. panoramic image capture · CPC title

• H04N23/66

  Remote control of cameras or camera parts, e.g. by remote control devices · CPC title