Retail shelf image processing and inventory tracking system

Patent metadata

Field	Value
Publication number	US-2023274410-A1
Application number	US-202217681507-A
Country	US
Kind code	A1
Filing date	Feb 25, 2022
Priority date	Feb 25, 2022
Publication date	Aug 31, 2023
Grant date	—

Abstract

The disclosed system and method relate to automatically detecting empty spaces on retail store shelves, identifying the missing product(s) and causing the space to be replenished or restocked. For example, stores may use shelf-mounted imaging devices to capture images of shelves across the aisle from the imaging devices. The images captured by the imaging devices may be pre-processed to de-warp, de-skew images and stitch together multiple images in order to retrieve an image that captures a full width of a shelf. The pre-processed images can then be used to detect products on the shelf, identify the detected products. An iterative projection algorithm or product fingerprint matching algorithm can be used to identify the products. When an incorrect product listing or an empty shelf space is encountered, a message may be sent to the store employee to remedy the issue.

First claim

What is claimed is:

1 . A method for identifying one or more products from an image, the method comprising:
receiving the image from an imaging device; detecting one or more products within the image; generating a fingerprint for a product, wherein the product is one of the one or more products; comparing the generated fingerprint with a plurality of reference fingerprints; determining one or more matches between the fingerprint and the plurality of reference fingerprints; using additional parameters associated with the image and the product, refining the one or more matches to a final match between the fingerprint and a final reference fingerprint; identifying a final reference product associated with the final reference fingerprint; and identifying the product as being the final reference product.

2 . The method of claim 1 , further comprising:
receiving one or more additional images from one or more additional imaging devices; processing the image and the one or more additional images to remove image skewing and image warping; and stitching the one or more additional images to the image.

3 . The method of claim 1 , wherein detecting the one or more product images within the image includes detecting the one or more images using a Cascade R-CNN object detection architecture.

4 . The method of claim 1 , wherein generating a fingerprint includes generating a vector representation of the product from the image based on a visual representation of the product on the image.

5 . The method of claim 1 , wherein the image is an image of a shelf within a retail store.

6 . The method of claim 5 , further comprising:
detecting a position in the image that is missing a missing product; analyzing one or more prior images, wherein one or more prior images are images of the shelf acquired earlier in time; generating a missing product fingerprint of a shelf product occupying the position in the one or more prior images; comparing the missing product fingerprint with the plurality of reference fingerprints; determining a missing product match between the missing product fingerprint and one of the plurality of reference fingerprints, wherein the one of the plurality of reference fingerprints is associated with a missing reference product; identifying the missing product as being the missing reference product; and sending a message to one or more user computing devices that the shelf is missing the missing product.

7 . The method of claim 5 , wherein each of the one or more reference fingerprints is associated with one of a plurality of reference products that is stocked by the retail store.

8 . The method of claim 7 , wherein the plurality of reference fingerprints and the plurality of reference products are stored in a database.

9 . The method of claim 1 , wherein one or more additional parameters includes: geolocation of the product, category of the product and product packaging elements.

10 . The method of claim 1 , wherein refining the one or more matches to a final match includes selecting a match among the one or more matches that is predicted to be the most accurate match by a deep neural network.

11 . A system for identifying one or more products from an image, the system comprising:
an imaging device; a computing system comprising:
a processor;
a memory communicatively connected to the processor which stores program instructions executable by the processor, wherein, when executed the program instructions cause the system to:
receive the image from the imaging device;
detect one or more products within the image;
generate a fingerprint for a product, wherein the product is one of the one or more products;
compare the generated fingerprint with a plurality of reference fingerprints;
determine one or more matches between the fingerprint and the plurality of reference fingerprints;
using additional parameters associated with the image and the product, refine the one or more matches to a final match between the fingerprint and a final reference fingerprint;
identify a final reference product associated with the final reference fingerprint; and
identify the product as being the final reference product.

12 . The system of claim 11 , wherein the program instructions further cause the computing system to:
receive one or more additional images from one or more additional imaging devices; process the image and the one or more additional images to remove image skewing and image warping; and stitch the one or more additional images to the image.

13 . The system of claim 11 , wherein detecting the one or more product images within the image includes detecting the one or more images using a Cascade R-CNN object detection architecture.

14 . The system of claim 11 , wherein generate a fingerprint includes generating a vector representation of the product from the image based on a visual representation of the product on the image.

15 . The system of claim 11 , wherein the image is an image of a shelf within a retail store.

16 . The system of claim 15 , wherein the program instructions further cause the computing system to:
detect a position in the image that is missing a missing product; analyze one or more prior images, wherein one or more prior images are images of the shelf acquired earlier in time; generate a missing product fingerprint of a shelf product occupying the position in the one or more prior images; compare the missing product fingerprint with the plurality of reference fingerprints; determine a missing product match between the missing product fingerprint and one of the plurality of reference fingerprints, wherein the one of the plurality of reference fingerprints is associated with a missing reference product; identify the missing product as being the missing reference product; and send a message to one or more user computing devices that the shelf is missing the missing product.

17 . The system of claim 15 , wherein each of the one or more reference fingerprints is associated with one of a plurality of reference products that is stocked by the retail store.

18 . The system of claim 10 , wherein one or more additional parameters includes: geolocation of the product, category of the product and product packaging elements.

19 . The system of claim 10 , wherein refining the one or more matches to a final match includes selecting a match among the one or more matches that is predicted to be the most accurate match by a deep neural network.

20 . A method for identifying one or more missing products from a shelf image, the method comprising:
receiving the shelf image from an imaging device, wherein the shelf image is an image of shelf within a retail store; detecting a position in the shelf image that is missing a missing product; analyzing one or more prior shelf images, wherein the one or more prior shelf images are images of the shelf acquired earlier in time; generating a product fingerprint of a shelf product occupying the position in the one or more prior shelf images; comparing the product fingerprint with a plurality of reference fingerprints; determining one or more product matches between the product fingerprint and the plurality of reference fingerprints; using additional parameters associated with the image and the missing product, refining the one or more product matches to a final product match between the product fingerprint and a final reference product fingerprint; identifying a final reference product as

Assignees

• Target Brands Inc

Inventors

• Patil Arun
• Mall Prakash
• Samal Snigdha
• Halder Soma
• Sethi Mohit
• Shivaram Swaroop

Classifications

• G06Q10/087 Primary

  Inventory or stock management, e.g. order filling, procurement or balancing against orders · CPC title

• G06V20/52

  Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69) · CPC title

• G06V10/74

  Image or video pattern matching; Proximity measures in feature spaces · CPC title

• G06V10/243

  by compensating for image skew or non-uniform image deformations · CPC title

• G06V10/16

  using multiple overlapping images; Image stitching · CPC title