Methods and systems for monitoring on-shelf inventory and detecting out of stock events

Patent metadata

Field	Value
Publication number	US-2023177458-A1
Application number	US-202117920692-A
Country	US
Kind code	A1
Filing date	Apr 14, 2021
Priority date	Apr 22, 2020
Publication date	Jun 8, 2023
Grant date	—

Abstract

Methods and systems for managing inventory at a retail facility include an image capture device having a field of view that includes a product display shelf of the retail facility, an electronic database that stores defined product display areas and assigned unique identifiers of the products on the product display shelves, and a computing device. The computing device obtains an image of a product display shelf, detects the individual packages of the different products captured in the image, identifies the products associated with the packages detected in the image by utilizing reference information stored in the database, determines whether one or more of the products on the product display shelf is associated with an out of stock event, and transmits an out of stock alert to an electronic device of a worker at the retail facility, notifying the worker to replenish the depleted products on the product display shelf.

First claim

What is claimed is:

1 . An inventory management system for a retail facility, the system comprising:
an image capture device having a field of view that includes a portion of a product display shelf of the retail facility having different products arranged thereon that are offered for sale to consumers at the retail facility; an electronic database that stores product display area data indicating reference product display areas associated each of the different products on the product display shelf, wherein each of the predefined product display areas is associated with individual packages representing one of the different products, and wherein a total number of the individual packages in the reference product display areas indicates a state of the reference product display areas when fully stocked; a computing device including a control circuit, the computing device being communicatively coupled to the image capture device and the electronic database, the control circuit being configured to:
obtain an image of the portion of the product display shelf captured by the image capture device;
correlate the obtained image to the product display area data stored in the electronic database and indicating the reference product display areas associated each of the different products on the product display shelf to:
identify both the products represented by the individual packages in the portion of the product display shelf in the image and the product display areas stocking the individual packages associated with the identified products; and
identify one or more product display areas where at least one of the individual packages of the identified productsis associated with an out of stock event; and

in response to an identification by the control circuit of one or more product display areas that the at least one of the individual packages of the identified products is associated with the out of stock event, transmit an out of stock alert to an electronic device of a worker at the retail facility, the out of stock alert identifying the one or more product display areas where the at least one of the individual packages of the identified products is associated with the out of stock event.

2 . The system of claim 1 , wherein the control circuit is configured to identify the products represented by the individual packages in the portion of the product display shelf in the image, the product display areas stocking the individual packages associated with the identified and one or more product display areas, where the individual packages of the identified products are associated with the out of stock event via a machine learning module comprising at least one trained neural network model that comprises:
an obstruction detection neural network model configured to detect a presence of one or more obstructions in the field of view of the image capture device; a gap detection neural network model configured to determine if there are empty spaces between the individual packages in their respective product display area; a segmentation detection neural network model configured to determine a percentage of empty spaces between the individual packages in their respective product display area; a messiness detection neural network model configured to determine whether the individual packages have shifted relative to their initially stocked position in their respective product display area; an object detection neural network model configured to determine counts of the individual packages in their respective product display area; a product recognition neural network model configured to recognize the products associated with the individual packages in their respective product display area; a label character recognition neural model configured to facilitate a verification, by the control circuit, that information on a label associated with the individual packages in their respective product display areas corresponds to product identifiers assigned in the electronic database to the identified products; and a binary classifier neural network model configured to detect presence or absence of the individual packages of the products in their respective product display area.

3 . The system of claim 2 , wherein the control circuit is further configured to:
receive a signal from the electronic device of the worker indicating, based on the worker inspecting the product display shelf, whether the out of stock alert was generated by the control circuit correctly or in error; if the signal received from the electronic device of the worker indicates that the out of stock alert was correctly generated by the control circuit, the control circuit is configured to cause the computing device to transmit an instruction to one or more workers at the retail facility to restock the product display shelf with additional individual packages of a product that was determined to be associated with the out of stock event; and if the signal received from the electronic device of the worker indicates that the out of stock alert was correctly generated by the control circuit, the control circuit is configured to retrain the at least one neural network model.

4 . The system of claim 1 , wherein the control circuit is further configured to determine a confidence level for the identification by the control circuit of one or more product display areas, where the individual packages of the identified products are associated with the out of stock event; and
if the confidence level is above a preset threshold, the control circuit communicates the out of stock alert to the electronic database.

5 . The system of claim 1 ,
wherein the image capture device comprises one or more of: a barcode (e.g., electronic product code (EPC), universal product code (UPC), European article number (EAN), global trade item number (GTIN) readers, quick response (QR) code readers, radio frequency identification (RFID) readers, stock keeping unit (SKU) readers, near field communication (NFC) readers, a digital camera, and a mobile electronic device; and wherein the image capture device is configured to capture images at predetermined intervals and to automatically send the captured images to the control circuit for analysis.

6 . The system of claim 5 , wherein the control circuit is configured to transmit the out of stock alert in response to obtaining a predetermined number of consecutive images determined by the control circuit to be indicative that the one or more of the product display areas have products associated with the out of stock event.

7 . The system of claim 1 , wherein, when the obstruction detection model detects the presence of one or more obstructions in the obtained image captured by the image capture device, the control circuit discards the obtained image without performing an out of stock analysis with respect to the obtained image.

8 . The system of claim 1 ,
wherein the image capture device includes at least two image capture devices having overlapping fields of view, such that the portion of the product display falls within the overlapping fields of view of the at least two image capture devices; and wherein the control circuit is configured to process the image captured by each of the at least two image capture devices and select the image determined by the control circuit, based on one or more quality factors, to be of higher quality.

9 . The system of claim 1 , wherein the control circuit is further configured to:
correlate the obtained image to the product display area data stored in the electronic database and indicating the reference product display areas associated each of the different products on the product display shelf to determine a total number of the

Assignees

• Walmart Apollo Llc

Inventors

• Kim Sung H
• Alvi Asim S
• Cook Jody P
• Gupta Kshitiz
• Jescheling Christopher R
• Kala Rahul S
• Michaelsamy Britto J
• Yardley Maxwell A
• Jones Matthew A
• Chandan Kumar
• Sirotkovic Jadran
• Arnold Mirko A
• Terlecky Peter M
• Wu Tirui

Classifications

• G06V10/82

  using neural networks · CPC title

• G06Q10/06311

  Scheduling, planning or task assignment for a person or group · CPC title

• G06Q10/087 Primary

  Inventory or stock management, e.g. order filling, procurement or balancing against orders · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title