Method for tracking placement of products on shelves in a store

Patent metadata

Field	Value
Publication number	US-12248908-B2
Application number	US-202217727044-A
Country	US
Kind code	B2
Filing date	Apr 22, 2022
Priority date	May 19, 2016
Publication date	Mar 11, 2025
Grant date	Mar 11, 2025

Abstract

One variation of a method for tracking placement of products in a store includes: accessing an image recorded by a mobile robotic system within a store; detecting a shelf in a region of the image; based on an address of the shelf, retrieving a list of products assigned to the shelf by a planogram of the store; retrieving a set of template images—from a database of template images—defining visual features of products specified in the list of products; extracting a set of features from the region of the image; determining that a unit of the product is mis-stocked on the shelf in response to deviation between the set of features and features in a template image, in the set of template images, representing the product; and in response to determining that the unit of the product is mis-stocked on the shelf, generating a restocking prompt for the product.

First claim

We claim:

1. A method comprising:
by a robotic system deployed in a store;
autonomously navigating along a first shelving structure within the store;
capturing a first image of a first shelving structure, within a field of view of an optical sensor arranged in the robotic system while occupying a first location at a first time; and
transmitting the first image to a computer system; and

by the computer system:
receiving the first image depicting the first shelving structure in the store from the robotic system at the first time;
identifying a first tag, arranged on the first shelving structure, depicted in a first region of the first image;
detecting a first set of features in the first region of the first image, the first set of features representing a first product descriptor;
accessing a first set of template features of a first product associated with the first product descriptor;
identifying a first slot, proximal the first tag, depicted in a second region of the first image;
detecting absence of the first product in the first slot based on absence of features analogous to the first set of template features in the second region of the first image;
accessing a first quantity of units of the first product assigned to the first slot by a product database; and
in response to detecting absence of the first product in the first slot, generating a first prompt to restock the second first slot with the first quantity of units of the first product.

2. The method of claim 1 :
wherein accessing the first set of template features of the first product comprises accessing a first set of template images depicting representative units of the first product in a range of orientations; and
wherein detecting absence of the first product in the first slot comprises:
detecting a second set of features from the second region of the first image; and
detecting absence of the first product in the first slot in response to absence of correlation between the second set of features and the first set of template images.

3. The method of claim 1 :
wherein accessing the first set of template features associated with the first product identifier comprises accessing the first set of template features comprising a first geometry and a first set of color features representative of the first product;
wherein detecting absence of the first product in the first slot of the first product in the first slot comprises:
identifying a first subregion in the second region of the first image bounding a first discrete object; and
detecting a third set of features comprising a second geometry and a second set of color features from the first subregion in the second region of the first image; and
calculating a composite score for correlations between the third set of features and the first set of template features; and
detecting absence of the first product in the first slot in response to the composite score falling below a threshold score.

4. The method of claim 1 , further comprising:
by the robotic system:
capturing a second image of a second shelving structure, within the field of view of the optical sensor while occupying a second location at a second time; and
transmitting the second image to the computer system; and

by the computer system:
receiving the second image depicting the second shelving structure in the store from the robotic system at the second time;
detecting a first shelf, on the second shelving structure, represented proximal a third region of the second image;
based on an address of the first shelf, accessing a first list of products assigned to the first shelf by the product database of the store;
accessing a first set of template images from a database of template images, each template image in the first set of template images comprising visual features of each product in the first list of products;
detecting a second set of features in the third region of the second image;
detecting improper stocking on the first shelf in response to deviation between second set of features and a first template image in the first set of template images; and
in response to detecting improper stocking on the first shelf, generating a second prompt to restock the first shelf on the second shelving structure.

5. The method of claim 4 , further comprising, by the computer system:
accessing a margin of the second product from a pricing database;
calculating a product value of the second product based on the margin of the second product and a current sale rate of the second product; and
generating a second prompt to restock the second slot:
in response to detecting a count of the second product in the second slot falling below a threshold count; and
in response to the second product value of the second product exceeding a threshold product value.

6. The method of claim 1 :
further comprising, by the computer system:
detecting an upper leading edge of a first shelf in the first image;
detecting a lower leading edge of the first shelf in the first image;
detecting the first tag between the upper leading edge of the first shelf and the lower leading edge of the first shelf depicted in the image; and
identifying a second tag, arranged on the first shelving structure, depicted in a third region of the first image; and

wherein identifying the first slot, proximal the first tag, depicted in the second region of the first image comprises:
identifying a first left edge of the first region proximal a corresponding left edge of the first tag;
identifying a second left edge of the third region proximal a corresponding left edge of the second tag; and
detecting the first slot interposed between the first left edge of the first region and the second left edge of the third region.

7. The method of claim 6 :
further comprising, by the computer system:
detecting a real offset between the first tag and the second tag from the first image;
detecting a second set of features in the third region of the first image, the second set of features representing a second product descriptor;
accessing a second set of template features of a second product associated with the second product descriptor;
identifying a second slot, proximal the second tag, depicted in a fourth region of the first image; and
accessing a relative slot position between the first slot and the second slot; and

wherein confirming presence of the first unit of the second product in the second slot comprises confirming presence of the first unit of the second product in the second slot:
in response to detecting features in the fourth region of the first image analogous to the second set of template features; and
in response to the real offset between the first tag and the second tag approximating the relative slot position between the first slot and the second slot.

8. The method of claim 1 :
wherein detecting the first set of features from the first region of the first image comprises detecting the first set of features from the first region of the first image, the first set of features representing a price value of the first product; and
further comprising, by the computer system:
reading the price value from the first tag depicted in the first image;
querying the product database associated with the store for a current list price assigned to the first product; and
in response to the price value differing from the current list price, flagging the first tag for correction.

9. The method of claim 1 :
wherein identifying the first tag, arranged on the first shelving structure, comprises identifying a first paper tag, arranged on the first shelving structure, depicted in the first region of the first image; and
wherein detecting

Assignees

• Simbe Robotics Inc

Inventors

• Bogolea Bradley
• Shah Mirza Akbar
• Vandegrift Lorin
• Fraser Luke
• Safi Jariullah
• Gee Jeffrey

Classifications

• G05D1/246

  using environment maps, e.g. simultaneous localisation and mapping [SLAM] · CPC title

• H04N23/80

  Camera processing pipelines; Components thereof · CPC title

• H04N25/615

  involving a transfer function modelling the optical system, e.g. optical transfer function [OTF], phase transfer function [PhTF] or modulation transfer function [MTF] · CPC title

• H04N23/698

  for achieving an enlarged field of view, e.g. panoramic image capture · CPC title

• H04N23/66

  Remote control of cameras or camera parts, e.g. by remote control devices · CPC title