Retail shelf image processing and inventory tracking system

Patent metadata

Field	Value
Publication number	US-12524868-B2
Application number	US-202217681507-A
Country	US
Kind code	B2
Filing date	Feb 25, 2022
Priority date	Feb 25, 2022
Publication date	Jan 13, 2026
Grant date	Jan 13, 2026

Abstract

The disclosed system and method relate to automatically detecting empty spaces on retail store shelves, identifying the missing product(s) and causing the space to be replenished or restocked. For example, stores may use shelf-mounted imaging devices to capture images of shelves across the aisle from the imaging devices. The images captured by the imaging devices may be pre-processed to de-warp, de-skew images and stitch together multiple images in order to retrieve an image that captures a full width of a shelf. The pre-processed images can then be used to detect products on the shelf, identify the detected products. An iterative projection algorithm or product fingerprint matching algorithm can be used to identify the products. When an incorrect product listing or an empty shelf space is encountered, a message may be sent to the store employee to remedy the issue.

First claim

What is claimed is:

1 . A method for identifying one or more products from an image, the method comprising:
receiving the image from an imaging device within a retail store; retrieving a location where the imaging device is affixed within the retail store; detecting one or more products within the image; generating a fingerprint for a product, wherein the product is one of the one or more products; comparing the generated fingerprint with a plurality of reference fingerprints; determining one or more matches between the fingerprint and the plurality of reference fingerprints; using additional parameters including product category data associated with the location of the imaging device and without mapping to planogram data, refining the one or more matches to a final match between the fingerprint and a final reference fingerprint, wherein the product category data include categories of products viewable by the imaging device based on the location where the imaging device is affixed within the retail store; identifying a final reference product associated with the final reference fingerprint; and identifying the product as being the final reference product.

2 . The method of claim 1 , further comprising:
receiving one or more additional images from one or more additional imaging devices; processing the image and the one or more additional images to remove image skewing and image warping; and stitching the one or more additional images to the image.

3 . The method of claim 1 , wherein detecting the one or more product images within the image includes detecting the one or more images using a Cascade R-CNN object detection architecture.

4 . The method of claim 1 , wherein generating a fingerprint includes generating a vector representation of the product from the image based on a visual representation of the product on the image.

5 . The method of claim 1 , further comprising:
identifying a missing product within the image by:
detecting a position in the image that is missing the missing product;
analyzing one or more prior images, wherein one or more prior images are images of the shelf acquired earlier in time;
generating a missing product fingerprint of a shelf product occupying the position in the one or more prior images;
comparing the missing product fingerprint with the plurality of reference fingerprints;
determining a missing product match between the missing product fingerprint and one of the plurality of reference fingerprints, wherein the one of the plurality of reference fingerprints is associated with a missing reference product;
identifying the missing product as being the missing reference product; and
sending a message to one or more user computing devices that the shelf is missing the missing product.

6 . The method of claim 1 , wherein each of the one or more reference fingerprints is associated with one of a plurality of reference products that is stocked by the retail store.

7 . The method of claim 1 , wherein the additional parameters further includes: product packaging elements.

8 . The method of claim 1 , wherein refining the one or more matches to a final match includes selecting a match among the one or more matches that is predicted to be the most accurate match by a deep neural network.

9 . The system of claim 1 , wherein the additional parameters further includes: product packaging elements.

10 . The method of claim 5 , wherein identifying the missing product is also completed without using a planogram.

11 . The method of claim 6 , wherein the plurality of reference fingerprints and the plurality of reference products are stored in a database.

12 . A system for identifying one or more products from an image, the system comprising:
an imaging device within a retail store; a computing system comprising:
a processor;
a memory communicatively connected to the processor which stores program instructions executable by the processor, wherein, when executed the program instructions cause the system to:
receive the image from the imaging device;
retrieve a location where the imaging device is affixed within the retail store;
detect one or more products within the image;
generate a fingerprint for a product, wherein the product is one of the one or more products;
compare the generated fingerprint with a plurality of reference fingerprints;
determine one or more matches between the fingerprint and the plurality of reference fingerprints;
using additional parameters including product category information associated with the location of the imaging device and without mapping to planogram data, refine the one or more matches to a final match between the fingerprint and a final reference fingerprint, wherein the product category data include categories of products viewable by the imaging device based on the location where the imaging device is affixed within the retail store;
identify a final reference product associated with the final reference fingerprint; and
identify the product as being the final reference product.

13 . The system of claim 12 , wherein the program instructions further cause the computing system to:
receive one or more additional images from one or more additional imaging devices; process the image and the one or more additional images to remove image skewing and image warping; and stitch the one or more additional images to the image.

14 . The system of claim 12 , wherein detecting the one or more product images within the image includes detecting the one or more images using a Cascade R-CNN object detection architecture.

15 . The system of claim 12 , wherein generate a fingerprint includes generating a vector representation of the product from the image based on a visual representation of the product on the image.

16 . The system of claim 12 , wherein the program instructions further cause the computing system to:
identifying a missing product within the image, including to:
detect a position in the image that is missing the missing product;
analyze one or more prior images, wherein one or more prior images are images of the shelf acquired earlier in time;
generate a missing product fingerprint of a shelf product occupying the position in the one or more prior images;
compare the missing product fingerprint with the plurality of reference fingerprints;
determine a missing product match between the missing product fingerprint and one of the plurality of reference fingerprints, wherein the one of the plurality of reference fingerprints is associated with a missing reference product;
identify the missing product as being the missing reference product; and
send a message to one or more user computing devices that the shelf is missing the missing product.

17 . The system of claim 12 , wherein each of the one or more reference fingerprints is associated with one of a plurality of reference products that is stocked by the retail store.

18 . The system of claim 12 , wherein refining the one or more matches to a final match includes selecting a match among the one or more matches that is predicted to be the most accurate match by a deep neural network.

19 . A method for identifying one or more missing products from a shelf image, the method comprising:
receiving the shelf image from an imaging device, wherein the shelf image is an image of shelf within a retail store; retrieving a location where the imaging device is affixed within the retail store; detecting a position in the shelf image that is missing a

Assignees

• Target Brands Inc

Inventors

• Patil Arun
• Mall Prakash
• Samal Snigdha
• Halder Soma
• Sethi Mohit
• Shivaram Swaroop

Classifications

• G06V20/52

  Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69) · CPC title

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title

• G06Q10/087 Primary

  Inventory or stock management, e.g. order filling, procurement or balancing against orders · CPC title

• G06V10/82

  using neural networks · CPC title

• G06V20/64

  Three-dimensional [3D] objects · CPC title