Image processing based methods and apparatus for planogram compliance

Patent metadata

Field	Value
Publication number	US-12136247-B2
Application number	US-202217825306-A
Country	US
Kind code	B2
Filing date	May 26, 2022
Priority date	Jun 24, 2021
Publication date	Nov 5, 2024
Grant date	Nov 5, 2024

Abstract

This application relates to automated processes for determining item placement compliance within retail locations. For example, a computing device may obtain an image of a fixture within a store. The image may be captured by a camera with a field of view directed at the fixture. The computing device may apply a segmentation process to the image to determine a portion of the image. Further, the computing device may determine a correlation between the portion of the image and each of a plurality of item image templates. Each item image template may include an image of an item the retail location sells in the retail location. The computing device may determine, based on the correlations, one of the plurality of item image templates and its corresponding item. The computing device may then determine whether the item should be located at the fixture based on a planogram.

First claim

What is claimed is:

1. A system comprising:
a computing device comprising at least one processor, wherein the computing device is configured to:
obtain an image;
determine a portion of the image based on applying a segmentation process to the image;
determine a correlation between the portion of the image and each of a plurality of item image templates;
determine one of the plurality of item image templates based on the correlations;
generate identification data associating the image to an item corresponding to the determined one of the plurality of item image templates;
store the identification data in a data repository;
determine an item placement location captured within the image;
obtain planogram data;
determine, based on the planogram data, whether the item is assigned to the item placement location; and
transmit a message that indicates the item and the item placement location.

2. The system of claim 1 , wherein the computing device is configured to:
determine that the item is not assigned to the item placement location.

3. The system of claim 1 , wherein determining the portion of the image comprises determining tag boundaries within the image.

4. The system of claim 1 , wherein determining the portion of the image comprises determining corners within the image.

5. The system of claim 4 , wherein determining the corners comprises applying a Harris Corner detection model to the image.

6. The system of claim 1 , wherein the computing device is configured to:
obtain image embeddings for each of the plurality of item image templates;
for each of the plurality of item image templates, determine a convolution of the image embeddings across the portion of the image; and
determine one of the plurality of item image templates based on the convolution of the images.

7. The system of claim 1 , wherein the computing device is configured to:
generate a plurality of keypoints based on the portion of the image;
determine a plurality of keypoint scores based on a matching of the plurality of keypoints to keypoints for each of the plurality of item image templates; and
determine one of the plurality of item image templates based on the keypoint scores.

8. The system of claim 1 , wherein the computing device is configured to:
apply a textual extraction process to the portion of the image data to extract image textual data;
apply a textual matching process to the extracted image textual data and to textual information associated with each of the plurality of item image templates to generate a text matching score; and
determine one of the plurality of item image templates based on the text matching scores.

9. A method comprising:
obtaining an image;
determining a portion of the image based on applying a segmentation process to the image;
determining a correlation between the portion of the image and each of a plurality of item image templates;
determining one of the plurality of item image templates based on the correlations;
generating identification data associating the image to an item corresponding to the determined one of the plurality of item image templates;
storing the identification data in a data repository;
determining an item placement location captured within the image;
obtaining planogram data;
determining, based on the planogram data, whether the item is assigned to the item placement location; and
transmitting a message that indicates the item and the item placement location.

10. The method of claim 9 , comprising:
determining that the item is not assigned to the item placement location.

11. The method of claim 9 , wherein determining the portion of the image comprises determining tag boundaries within the image.

12. The method of claim 9 , wherein determining the portion of the image comprises determining corners within the image.

13. The method of claim 12 , wherein determining the corners comprises applying a Harris Corner detection model to the image.

14. The method of claim 9 comprising:
obtaining image embeddings for each of the plurality of item image templates;
for each of the plurality of item image templates, determining a convolution of the image embeddings across the portion of the image; and
determining one of the plurality of item image templates based on the convolution of the images.

15. The method of claim 9 comprising:
generating a plurality of keypoints based on the portion of the image;
determining a plurality of keypoint scores based on a matching of the plurality of keypoints to keypoints for each of the plurality of item image templates; and
determining one of the plurality of item image templates based on the keypoint scores.

16. The method of claim 9 comprising:
applying a textual extraction process to the portion of the image data to extract image textual data;
applying a textual matching process to the extracted image textual data and to textual information associated with each of the plurality of item image templates to generate a text matching score; and
determining one of the plurality of item image templates based on the text matching scores.

17. A non-transitory computer readable medium having instructions stored thereon, wherein the instructions, when executed by at least one processor, cause a device to perform operations comprising:
obtaining an image;
determining a portion of the image based on applying a segmentation process to the image;
determining a correlation between the portion of the image and each of a plurality of item image templates;
determining one of the plurality of item image templates based on the correlations;
generating identification data associating the image to an item corresponding to the determined one of the plurality of item image templates;
storing the identification data in a data repository;
determining an item placement location captured within the image;
obtaining planogram data;
determining, based on the planogram data, whether the item is assigned to the item placement location; and
transmitting a message that indicates the item and the item placement location.

Assignees

• Walmart Apollo Llc

Inventors

• Karmakar Somedip
• Chakraborty Souradip
• Mishra Abhishek

Classifications

• G06T7/11

  Region-based segmentation · CPC title

• G06T7/73

  using feature-based methods · CPC title

• G06V10/54

  relating to texture · CPC title

• G06V10/46

  Descriptors for shape, contour or point-related descriptors, e.g. scale invariant feature transform [SIFT] or bags of words [BoW]; Salient regional features (colour feature extraction G06V10/56) · CPC title

• G06V10/761

  Proximity, similarity or dissimilarity measures · CPC title