Method for stock keeping in a store with fixed cameras

Patent metadata

Field	Value
Publication number	US-12177387-B2
Application number	US-202318099901-A
Country	US
Kind code	B2
Filing date	Jan 20, 2023
Priority date	May 19, 2016
Publication date	Dec 24, 2024
Grant date	Dec 24, 2024

Abstract

One variation of a method for stock keeping in a store includes: accessing an image captured by a fixed camera within the store; retrieving a field of view of the fixed camera; estimating a segment of an inventory structure in the store depicted in the image based on a projection of the field of view onto a planogram of the store; identifying a set of slots within the inventory structure segment; retrieving a product model representing a set of visual characteristics of a product type assigned to a slot, in the set of slots, by the planogram; extracting a constellation of features from the image; if the constellation of features approximates the set of visual characteristics in the product model, detecting presence of a product unit of the product type occupying the inventory structure segment; and representing presence of the product unit, occupying the inventory structure segment, in a realogram.

First claim

We claim:

1. A method comprising:
accessing an image captured by an optical sensor, arranged within a store, at a first time;
detecting a segment of an inventory structure in the store in a first region of the image;
identifying a set of slots within the segment of the inventory structure;
accessing a first product model representing a first set of visual characteristics of a first product type assigned to a first slot, in the set of slots, by a graphical representation of the store;
detecting a first set of features in a subregion of the image;
detecting presence of a first product unit of the first product type occupying the segment of the inventory structure at the first time in response to the first set of features approximating the first set of visual characteristics;
detecting a first shelf tag in the image;
interpreting a set of possible product identifiers from the first shelf tag; and
flagging the first shelf tag for inspection in response to the set of possible product identifiers excluding a first product identifier of the first product type.

2. The method of claim 1 :
further comprising:
accessing a location and an orientation of the optical sensor within a coordinate system of the store at the first time;
retrieving a set of imaging properties of the optical sensor;
calculating a position of a focal axis of the optical sensor within the coordinate system of the store based on the location and the orientation of the optical sensor within the coordinate system;
calculating a volumetric imaging boundary extending along the focal axis of the optical sensor based on the set of imaging properties of the optical sensor; and
storing the volumetric imaging boundary as a geometry of the field of view of the optical sensor at the first time; and

wherein detecting the segment of the inventory structure in the store in the first region of the image comprises:
projecting the volumetric imaging boundary onto the graphical representation referenced to the coordinate system of the store; and
calculating the segment of the inventory structure based on an intersection of the volumetric imaging boundary and a representation of the inventory structure in the graphical representation of the store.

3. The method of claim 1 , further comprising:
detecting a set of inventory structure features in the image;
identifying a set of inventory structures depicted in the image based on the set of inventory structure features;
retrieving locations and orientations of the set of inventory structures within a coordinate system of the store;
calculating a field of view of the optical sensor within the coordinate system of the store:
based on locations and orientations of the set of inventory structures within the coordinate system of the store; and
based on positions of the set of inventory structure features in the image; and

wherein detecting the segment of the inventory structure in the store in the first region of the image comprises:
retrieving a geometry of the field of view of the optical sensor at the first time;
projecting the geometry of the field of view onto the graphical representation of the store referenced to the coordinate system of the store; and
calculating the segment of the inventory structure in the store based on an intersection of the geometry of the field of view and a representation of the inventory structure in the graphical representation of the store.

4. The method of claim 1 , further comprising:
extracting a slot boundary of a second slot, in the set of slots, in the segment of the inventory structure from the graphical representation of the store, the graphical representation corresponding to a target stock condition of the store;
projecting the second slot boundary onto the image;
retrieving a second product model representing a second set of visual characteristics of a second product type assigned to the inventory structure by the graphical representation of the store;
detecting a second object within the second slot boundary in the image;
detecting a second set of features from a second subregion of the image depicting the second object;
detecting presence of a second product unit of the second product type occupying the second slot in the inventory structure at the first time in response to the second set of features approximating the second set of visual characteristics; and
flagging the second product unit for removal from the second slot in the inventory structure in response to the second product type deviating from a third product type assigned to the second slot in the inventory structure by the graphical representation of the store.

5. The method of claim 4 , further comprising:
extracting a second slot boundary of a second slot, in the set of slots, in the segment of the inventory structure from the graphical representation of the store;
projecting the second slot boundary onto the image; and
in response to detecting absence of objects within the second slot boundary in the image:
representing an out-of-stock condition, in the second slot in the inventory structure, in a second graphical representation of the store corresponding to a current stock condition of the store; and
flagging the second slot in the inventory structure for restocking with product units of a second product type assigned to the second slot by the graphical representation of the store.

6. The method of claim 1 , further comprising:
accessing a second image captured by a second optical sensor, arranged within the store and offset from the optical sensor, at approximately the first time;
detecting a second segment of the inventory structure in the store in a second region of the second image;
identifying a second set of slots within the second segment of the inventory structure;
retrieving a second product model representing a second set of visual characteristics of a second product type assigned to a second slot, in the second set of slots, by the graphical representation of the store;
extracting a second set of features from the second image;
detecting presence of a second product unit of the second product type occupying the second segment of the inventory structure at the first time in response to the second set of features approximating the second set of visual characteristics;
detecting a second shelf tag in the second image;
interpreting a second set of possible product identifiers from the second shelf tag; and
confirming accuracy of the second shelf tag in response to the second set of possible product identifiers comprising a second product identifier of the second product type.

7. The method of claim 6 , further comprising:
accessing a third image:
captured by a robotic system during a scan cycle intersecting the first time; and
depicting a third segment of the inventory structure between the segment of the inventory structure and the second segment of the inventory structure;

reading a third product identifier of a third product type from a third shelf tag detected in the third photographic image;
retrieving a third product model representing a third set of visual characteristics of the third product type;
based on the third shelf tag detected in the third photographic image, locating a third region in the third photographic image depicting a third slot in the inventory structure;
extracting a third set of features from the third region of the third photographic image; and
detecting presence of a third product unit of the third product type occupying the third slot in the third segment of the inventory structure during the scan cycle in response to the third set of features approximating the third set of visual characteristics represented in the third product model; and
aggregating pre

Assignees

• Simbe Robotics Inc

Inventors

• Bogolea Bradley
• Shah Mirza Akbar
• Vandegrift Lorin
• Fraser Luke
• Safi Jariullah
• Gee Jeffrey
• Tiwari Durgesh

Classifications

• G05D1/223

  Command input arrangements on the remote controller, e.g. joysticks or touch screens · CPC title

• G05D1/246

  using environment maps, e.g. simultaneous localisation and mapping [SLAM] · CPC title

• G05D1/249

  from positioning sensors located off-board the vehicle, e.g. from cameras · CPC title

• G06V10/987

  with the intervention of an operator · CPC title

• G06V10/751

  Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching · CPC title