Systems and methods of defining and identifying product display areas on product display shelves

Patent metadata

Field	Value
Publication number	US-2023252407-A1
Application number	US-202117920610-A
Country	US
Kind code	A1
Filing date	Apr 16, 2021
Priority date	Apr 22, 2020
Publication date	Aug 10, 2023
Grant date	—

Abstract

Methods and systems for managing inventory at a retail facility include an image capture device having a field of view that includes a product display shelf of the retail facility, an electronic database that stores a planogram of product display shelves at the retail facility and a computing device. The computing device obtains an image of a product display shelf, detects the individual packages of each of the different products captured in the image, defines different product display areas for the different products on the product display shelf captured in the image, and generates virtual boundary lines that surround each of the defined product display areas. The computing device also determines an identity of a product contained located in each defined product display area, and associates the virtual boundary lines surrounding each defined product display area with an identifier unique to the product contained in the defined product display area.

First claim

1 . An inventory management system of a retail facility, the system comprising:
an image capture device having a field of view that includes a product display shelf of the retail facility having different products that are offered for sale to consumers at the retail facility arranged therein; an electronic database that stores a planogram of product display shelves at the retail facility, the planogram indicating physical locations of the products arranged on the product display shelf; and a computing device including a control circuit, the computing device being communicatively coupled to the image capture device and the electronic database, the control circuit being configured to: obtain an image of a product display shelf captured by the image capture device; detect individual packages of each of the different products captured in the image; define a plurality of product display areas for the different products on the product display shelf captured in the image, wherein each of the product display areas includes the individual packages representing one of the different products; generate a plurality of virtual boundary lines in the image, wherein each of the virtual boundary lines surrounds one of the defined product display areas; correlate the image having each of the virtual boundary lines surrounding one of the defined product display areas to the planogram stored in the electronic database in order to determine an identity of a product contained in the individual packages located in each one of the defined product display areas; and assign product identifiers to the virtual boundary lines in the image, wherein a product identifier assigned to a virtual boundary line surrounding a product display area defined by the control circuit uniquely identifies the product contained in the individual packages located in the product display area surrounded by the virtual boundary line.

2 . The system of claim 1 , wherein the control circuit is further configured to cause the computing device to transmit a signal to the electronic database, the signal including electronic data for storage in the electronic database, the electronic data in the signal indicating physical locations of each of the product display areas defined by the control circuit, a perimeter of the product display areas as defined by the virtual boundary lines generated by the control circuit, and the product identifiers assigned to virtual boundary lines of the product display areas.

3 . The system of claim 1 , wherein the control circuit is configured to process the obtained image and to extract metadata from the image indicative of a physical location of at least a portion of the product display shelf depicted in the obtained image.

4 . The system of claim 1 , wherein the control circuit is further configured to:
correlate the image having each of the virtual boundary lines surrounding one of the defined product display areas to the planogram stored in the electronic database via a machine learning module comprising a neural network model; and execute the neural network model to generate the plurality of virtual boundary lines; wherein the neural network model is trained using the planogram and the virtual boundary lines and the product identifiers associated with the product display areas defined by the control circuit.

5 . The system of claim 1 , wherein the image capture device comprises one or more of: barcode readers, quick response (QR) code readers, radio frequency identification (RFID) readers, stock keeping unit (SKU) readers, near field communication (NFC) readers, a digital camera, and a mobile electronic device.

6 . The system of claim 1 ,
wherein the image capture device includes at least two image capture devices having overlapping fields of view, such that at least one portion of the product display shelf falls within the overlapping fields of view of the at least two image capture devices; and wherein the control circuit is configured to process the image captured by each of the at least two image capture devices and select the image determined by the control circuit, based on one or more quality factors, to be of higher quality.

7 . The system of claim 1 , wherein the product display shelf further comprises a plurality of dividers separating the product display areas, wherein each one of the product display areas having at least one adjacent product display area is separated from the at least one adjacent product display area by at least one of the dividers.

8 . The system of claim 7 , wherein the control circuit is configured to detect the dividers in the obtained image of the product display shelf, and to interpret the at least one of the dividers separating the at least one adjacent product display area as at least one boundary of the at least one adjacent product display area separated by the at least one of the dividers.

9 . The system of claim 7 ,
further comprising at least one product label readable by the image capture device and configured to be attached to at least one of the dividers, the product display shelf, and the products on the product display shelf; and wherein the control circuit is further configured to:
detect the at least one product label in the obtained image of the product display shelf from the image capture device;
recognize at least one of price information, product location information, and product identifier information on the detected at least one product label; and
correlate the at least one of the price information, product location information, and product identifier information recognized on the detected at least one product label to the planogram and the virtual boundary lines and the product identifiers associated with the product display areas defined by the control circuit in order to measure accuracy of the control circuit in determining the identity of the product contained in the individual packages located in each one of the defined product display areas.

10 . The system of claim 1 , wherein the product display shelf includes a product support surface, the product support surface being oriented at an angle, such that a rear portion of the product display shelf is positioned higher than a front portion of the product display shelf, the product support surface comprising a plurality of rollers configured to rotate and permit the individual packages arranged on the product display shelf to move via gravity in a direction toward the front portion of the product display shelf to fill in an empty space caused by removal of one of the individual packages from the product support surface.

11 . A method of managing inventory of a retail facility, the method comprising:
providing an image capture device having a field of view that includes a product display shelf of the retail facility having different products that are offered for sale to consumers at the retail facility arranged therein; providing an electronic database that stores a planogram of product display shelves at the retail facility, the planogram indicating physical locations of the products arranged on the product display shelf; and providing a computing device including a control circuit, the computing device being communicatively coupled to the image capture device and the electronic database; by the control circuit of the computing device:
obtaining an image of a product display shelf captured by the image capture device;
detecting individual packages of each of the different products captured in the image;
defining a plurality of product display areas for the different products on the product display shelf captured in the image, wherein each of the product display areas includes the ind

Assignees

• Walmart Apollo Llc

Inventors

• Kim Sung H
• Alvi Asim S
• Cook Jody P
• Gupta Kshitiz
• Jescheling Christopher R
• Jones Matthew A
• Kala Rahul S
• Michaelsamy Britto J
• Yardley Maxwell A
• Chandan Kumar
• Sirotkovic Jadran
• Arnold Mirko A
• Terlecky Peter M
• Wu Tirui

Classifications

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06Q10/087 Primary

  Inventory or stock management, e.g. order filling, procurement or balancing against orders · CPC title

• A47F1/08

  dispensing from bottom · CPC title

• G06V10/82

  using neural networks · CPC title