Method for stock keeping in a store with fixed cameras

Patent metadata

Field	Value
Publication number	US-11593755-B2
Application number	US-202016880582-A
Country	US
Kind code	B2
Filing date	May 21, 2020
Priority date	May 19, 2016
Publication date	Feb 28, 2023
Grant date	Feb 28, 2023

Abstract

A method for stock keeping in a store includes: accessing an image captured by a fixed camera within the store; retrieving a field of view of the fixed camera; estimating a segment of an inventory structure in the store depicted in the image based on a projection of the field of view onto a planogram of the store; identifying a set of slots within the inventory structure segment; retrieving a product model representing a set of visual characteristics of a product type assigned to a slot, in the set of slots, by the planogram; extracting a constellation of features from the image; if the constellation of features approximates the set of visual characteristics in the product model, detecting presence of a product unit of the product type occupying the inventory structure segment; and representing presence of the product unit, occupying the inventory structure segment, in a realogram.

First claim

We claim:

1. A method for stock keeping in a store comprising:
accessing a photographic image captured by a fixed camera, arranged within the store, at a first time;
retrieving a geometry of a field of view of the fixed camera at the first time;
estimating a shelving segment of an inventory structure in the store depicted in the photographic image based on a projection of the geometry of the field of view onto a planogram of the store;
identifying a set of slots within the shelving segment of the inventory structure;
retrieving a first product model representing a first set of visual characteristics of a first product type assigned to a first slot, in the set of slots, by the planogram;
extracting a first constellation of features from the photographic image;
detecting presence of a first product unit of the first product type occupying the shelving segment of the inventory structure at the first time in response to the first constellation of features approximating the first set of visual characteristics represented in the first product model; and
representing presence of the first product unit, occupying the shelving segment of the inventory structure at the first time, in a realogram.

2. The method of claim 1 :
wherein retrieving the geometry of the field of view of the fixed camera at the first time comprises:
accessing a location and an orientation of the fixed camera within a coordinate system of the store at the first time;
retrieving a set of imaging properties of the fixed camera;
calculating a position of a focal axis of the fixed camera within the coordinate system of the store based on the location and the orientation of the fixed camera within the coordinate system;
calculating a volumetric imaging boundary extending along the focal axis of the fixed camera based on the set of imaging properties of the fixed camera; and
storing the volumetric imaging boundary as the geometry of the field of view of the fixed camera at the first time; and

wherein estimating the shelving segment of the inventory structure in the store depicted in the photographic image comprises:
projecting the volumetric imaging boundary onto the planogram referenced to the coordinate system of the store; and
calculating the shelving segment of the inventory structure based on an intersection of the volumetric imaging boundary and a representation of the inventory structure in the planogram.

3. The method of claim 1 , wherein retrieving the geometry of the field of view of the fixed camera at the first time comprises:
detecting a set of inventory structure features in the photographic image;
identifying a set of inventory structures depicted in the photographic image based on the set of inventory structure features;
retrieving locations and orientations of the set of inventory structures within a coordinate system of the store; and
calculating a field of view of the fixed camera within the coordinate system of the store based on locations and orientations of the set of inventory structures within the coordinate system of the store and based on positions of the set of inventory structure features in the photographic image.

4. The method of claim 1 :
further comprising:
detecting the shelving segment of the inventory structure in a first region of the photographic image; and
detecting a set of objects in the region of the photographic image;

wherein extracting the first constellation of features from the photographic image and detecting presence of the first product unit of the first product type comprises, for each object in the set of objects:
extracting a constellation of features from a subregion of the photographic image depicting the object;
identifying the object as a product unit, in a first subset of product units, of the first product type in response to the constellation of features approximating the first set of visual characteristics represented in the first product model; and
identifying the object as a product unit, in a second subset of product units, of other than the first product type in response to the constellation of features differing from the first set of visual characteristics represented in the first product model; and

wherein representing presence of the first product unit in the realogram comprises representing presence of the first subset of product units, of the first product type and occupying the shelving segment of the inventory structure, in the realogram.

5. The method of claim 4 , further comprising:
retrieving a second product model representing a second set of visual characteristics of a second product type assigned to a second slot, in the set of slots, by the planogram;
for each object in a second subset of objects:
identifying the object as a product unit, in a third subset of product units, of the second product type in response to a constellation of features, extracted from a subregion of the photographic image depicting the object, approximating the second set of visual characteristics represented in the second product model; and
identifying the object as a product unit, in a fourth subset of product units, of other than the second product type in response to the constellation of features differing from the second set of visual characteristics represented in the second product model;

representing presence of the third subset of product units, of the second product type and occupying the shelving segment of the inventory structure, in the realogram; and
flagging the shelving segment of the inventory structure for restocking with product units of the second product type in response to the third subset of product units comprising a null set.

6. The method of claim 1 :
further comprising:
extracting a first slot boundary of the first slot in the shelving segment of the inventory structure from the planogram;
projecting the first slot boundary onto the photographic image;

wherein extracting the first constellation of features from the photographic image comprises:
detecting a first object within the first slot boundary projected onto the photographic image; and
extracting the first constellation of features from a first region of the photographic image depicting the first object;

wherein detecting presence of the first product unit of the first product type occupying the shelving segment of the inventory structure at the first time comprises detecting presence of the first product unit of the first product type occupying the first slot in the inventory structure at the first time in response to the first constellation of features approximating the first set of visual characteristics represented in the first product model; and
wherein representing presence of the first product unit in the realogram comprises representing presence of the first product unit, occupying the first slot in the inventory structure, in the realogram.

7. The method of claim 6 , further comprising:
extracting a second slot boundary of a second slot, in a set of slots, in the shelving segment of the inventory structure from the planogram;
projecting the second slot boundary onto the photographic image;
retrieving a second product model representing a second set of visual characteristics of a second product type assigned to the inventory structure by the planogram;
detecting a second object within the second slot boundary projected onto the photographic image;
extracting a second constellation of features from a second region of the photographic image depicting the second object;
detecting presence of a second product unit of the second product type occupying the second slot in the inventory structure at the first time in response to the second constellation of features approximating the second set of visual characteri

Assignees

• Simbe Robotics Inc

Inventors

• Bogolea Bradley
• Shah Mirza Akbar
• Vandegrift Lorin
• Fraser Luke
• Safi Jariullah
• Gee Jeffrey
• Tiwari Durgesh

Classifications

• H04N23/90

  Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums · CPC title

• G06F18/22

  Matching criteria, e.g. proximity measures · CPC title

• H04N23/60

  Control of cameras or camera modules · CPC title

• G06V10/751

  Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching · CPC title

• G06V2201/06

  Recognition of objects for industrial automation · CPC title