Method for tracking placement of products on shelves in a store

Patent metadata

Field	Value
Publication number	US-10467587-B2
Application number	US-201715600527-A
Country	US
Kind code	B2
Filing date	May 19, 2017
Priority date	May 19, 2016
Publication date	Nov 5, 2019
Grant date	Nov 5, 2019

Abstract

One variation of a method for tracking placement of products in a store includes: accessing an image recorded by a mobile robotic system within a store; detecting a shelf in a region of the image; based on an address of the shelf, retrieving a list of products assigned to the shelf by a planogram of the store; retrieving a set of template images—from a database of template images—defining visual features of products specified in the list of products; extracting a set of features from the region of the image; determining that a unit of the product is mis-stocked on the shelf in response to deviation between the set of features and features in a template image, in the set of template images, representing the product; and in response to determining that the unit of the product is mis-stocked on the shelf, generating a restocking prompt for the product.

First claim

We claim:

1. A method for tracking placement of products on shelves in a store comprising:
accessing a first image comprising visual data recorded by a mobile robotic system at a first time, the first image associated with a first location and a first orientation of the mobile robotic system within the store at the first time;
detecting a first shelf represented proximal a first region of the first image;
determining an address of the first shelf based on the first location, the first orientation, and a vertical position of the first shelf within the first image;
based on the address of the first shelf, retrieving a first list of products assigned to the first shelf by a planogram of the store;
retrieving a first set of template images from a database of template images, each template image in the first set of template images comprising visual features of a product specified in the first list of products;
extracting a first set of features from the first region of the first image;
confirming presence of a unit of a first product, in the first list of products, on the first shelf in response to correlation between features in the first set of features and features in a first template image, in the first set of template images, representing the first product;
determining that a unit of the first product is improperly stocked on the first shelf in response to deviation between features in the first set of features and features in the first template image; and
in response to determining that the unit of the first product is improperly stocked on the first shelf, generating a first restocking prompt for the first product on the first shelf.

2. The method of claim 1 :
wherein extracting the first set of features from the first region of the first image comprises:
from the first region of the first image, detecting a set of discrete objects arranged across the first shelf;
identifying a first subregion in the first region of the first image corresponding to a first discrete object, in the set of discrete objects, located at a first position on the first shelf; and
extracting the first set of features from the first subregion in the first region of the first image; and

wherein confirming presence of the unit of the first product on the first shelf comprises identifying the first discrete object as the unit of the first product in response to correlation between features in the first set of features and features in the first template image representing the first product.

3. The method of claim 2 :
further comprising:
identifying a second subregion in the first region of the first image corresponding to a second discrete object, in the set of discrete objects, located at a second position on the first shelf;
extracting a second set of features from the second subregion in the first region of the first image;
identifying the second discrete object as a unit of a second product, in the first list of products, in response to correlation between features in the second set of features and features in a second template image, in the first set of template images, representing the second product; and
in response to identifying the first discrete object as the unit of the first product and identifying the second discrete object as the unit of the second product, detecting improper arrangement of the unit of the first product and the unit of the second product in response to a lateral offset between the first position and the second position deviating from a target lateral offset assigned to the first product and the second product on the first shelf by the planogram; and

wherein generating the first restocking prompt for the first product on the first shelf comprises generating the first restocking prompt specifying correction of relative positions of the unit of the first product and the unit of the second product in response to detecting improper arrangement of the first product and the second product on the first shelf.

4. The method of claim 1 :
wherein confirming presence of the unit of a first product on the first shelf comprises confirming presence of a set of discrete units of the first product on the first shelf in response to discrete clusters of features in the first set of features exhibiting strong correlation to features in the first template image representing the first product;
further comprising:
generating a count of units of the first product in the set of discrete units of the first product on the first shelf;
confirming proper stocking of the first product on the first shelf in response to the count equaling a quantity of facings of the first product assigned to the first shelf by the planogram; and
detecting improper stocking of the first product on the first shelf in response to the count deviating from the quantity of facings of the first product assigned to the first shelf by the planogram; and

wherein generating the first restocking prompt for the first product on the first shelf comprises generating the first restocking prompt specifying correction of facings of the first product on the first shelf in response to detecting improper stocking of the first product on the first shelf.

5. The method of claim 1 , further comprising:
detecting a second shelf represented in a second region of the first image below the first region;
determining an address of the second shelf based on the first location, the first orientation, and a vertical position of the second shelf within the first image;
based on the address of the second shelf, retrieving a second list of products assigned to the second shelf by the planogram;
retrieving a second set of template images from the database of template images, each template image in the second set of template images comprising visual features of a product specified in the second list of products;
extracting a second set of features from the second region of the first image;
confirming presence of a unit of a second product, in the second list of products, on the second shelf in response to a first degree of correlation between features in the second set of features and features in a second template image, in the second set of template images, representing the second product;
determining that a unit of the second product is improperly stocked on the second shelf in response to a second degree of correlation, less than the first degree, between features in the second set of features and features in the second template image;
in response to determining that the unit of the second product is improperly stocked on the first shelf, generating a second restocking prompt for the second product on the second shelf; and
compiling the first restocking prompt and the second restocking prompt into a global electronic restocking list.

6. The method of claim 1 :
wherein confirming presence of the unit of the first product on the first shelf comprises:
matching features, in the first set of features extracted from the first region of the first image, to a first group of features defined in the first template image of the first product;
confirming absence of a second group of features, defined in the first template image, from the first set of features extracted from the first region of the first image; and
detecting orientation of the unit of the first product on the shelf in other than a target orientation in response to the first set of features comprising features matching the first group of features and omitting the second group of features in the first template image representing the first product; and

wherein generating the first restocking prompt for the first product on the first shelf comprises generating the first restocking prompt to correct orientation of the unit of the first product on the first shelf in response

Assignees

• Simbe Robotics Inc

Inventors

• Bogolea Bradley
• Shah Mirza Akbar
• Vandegrift Lorin
• Fraser Luke
• Safi Jariullah
• Gee Jeffrey

Classifications

• H04N25/615

  involving a transfer function modelling the optical system, e.g. optical transfer function [OTF], phase transfer function [PhTF] or modulation transfer function [MTF] · CPC title

• H04N23/698

  for achieving an enlarged field of view, e.g. panoramic image capture · CPC title

• G06F18/22

  Matching criteria, e.g. proximity measures · CPC title

• G06F18/21

  Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation · CPC title

• H04N23/66

  Remote control of cameras or camera parts, e.g. by remote control devices · CPC title