Image processing based methods and apparatus for planogram compliance

Patent metadata

Field	Value
Publication number	US-2022415012-A1
Application number	US-202217825306-A
Country	US
Kind code	A1
Filing date	May 26, 2022
Priority date	Jun 24, 2021
Publication date	Dec 29, 2022
Grant date	—

Abstract

This application relates to automated processes for determining item placement compliance within retail locations. For example, a computing device may obtain an image of a fixture within a store. The image may be captured by a camera with a field of view directed at the fixture. The computing device may apply a segmentation process to the image to determine a portion of the image. Further, the computing device may determine a correlation between the portion of the image and each of a plurality of item image templates. Each item image template may include an image of an item the retail location sells in the retail location. The computing device may determine, based on the correlations, one of the plurality of item image templates and its corresponding item. The computing device may then determine whether the item should be located at the fixture based on a planogram.

First claim

What is claimed is:

1 . A system comprising:
a computing device comprising at least one processor, where the computing device is configured to:
obtain an image;
determine a portion of the image based on applying a segmentation process to the image;
determine a correlation between the portion of the image and each of a plurality of item image templates;
determine one of the plurality of item image templates based on the correlations;
generate identification data associating the image to an item corresponding to the determined one of the plurality of item image templates; and
store the identification data in a data repository.

2 . The system of claim 1 , wherein the computing device is configured to:
determine an item placement location captured within the image; obtain planogram data; determine, based on the planogram data, whether the item is assigned to the item placement location.

3 . The system of claim 2 , wherein the computing device is configured to:
determine that the item is not assigned to the item placement location; and transmit a message that indicates the item and the item placement location.

4 . The system of claim 1 , wherein determining the portion of the image comprises determine tag boundaries within the image.

5 . The system of claim 1 , wherein determining the portion of the image comprises determining corners within the image.

6 . The system of claim 5 , wherein determining the corners comprises applying a Harris Corner detection model to the image.

7 . The system of claim 1 , wherein the computing device is configured to:
obtain image embeddings for each of the plurality of item image templates; for each of the plurality of item image templates, determine a convolution of the image embeddings across the portion of the image; and determine one of the plurality of item image templates based on the convolution of the images.

8 . The system of claim 1 , wherein the computing device is configured to:
generate a plurality of keypoints based on the portion of the image; determine a plurality of keypoint scores based on a matching of the plurality of keypoints to keypoints for each of the plurality of item image templates; and determine one of the plurality of item image templates based on the keypoint scores.

9 . The system of claim 1 , wherein the computing device is configured to:
apply a textual extraction process to the portion of the image data to extract image textual data; apply a textual matching process to the extracted image textual data and to textual information associated with each of the plurality of item image templates to generate a text matching score; and determine one of the plurality of item image templates based on the text matching scores.

10 . A method comprising:
obtaining an image; determining a portion of the image based on applying a segmentation process to the image; determining a correlation between the portion of the image and each of a plurality of item image templates; determining one of the plurality of item image templates based on the correlations; generating identification data associating the image to an item corresponding to the determined one of the plurality of item image templates; and storing the identification data in a data repository.

11 . The method of claim 10 comprising:
determining an item placement location captured within the image;
obtaining planogram data;
determining, based on the planogram data, whether the item is assigned to the item placement location.

12 . The method of claim 11 comprising:
determining that the item is not assigned to the item placement location; and
transmitting a message that indicates the item and the item placement location.

13 . The method of claim 10 , wherein determining the portion of the image comprises determine tag boundaries within the image.

14 . The method of claim 10 , wherein determining the portion of the image comprises determining corners within the image.

15 . The method of claim 14 , wherein determining the corners comprises applying a Harris Corner detection model to the image.

16 . The method of claim 10 comprising:
obtaining image embeddings for each of the plurality of item image templates;
for each of the plurality of item image templates, determining a convolution of the image embeddings across the portion of the image; and
determining one of the plurality of item image templates based on the convolution of the images.

17 . The method of claim 10 comprising:
generating a plurality of keypoints based on the portion of the image;
determining a plurality of keypoint scores based on a matching of the plurality of keypoints to keypoints for each of the plurality of item image templates; and
determining one of the plurality of item image templates based on the keypoint scores.

18 . The method of claim 10 comprising:
applying a textual extraction process to the portion of the image data to extract image textual data;
applying a textual matching process to the extracted image textual data and to textual information associated with each of the plurality of item image templates to generate a text matching score; and
determining one of the plurality of item image templates based on the text matching scores.

19 . A non-transitory computer readable medium having instructions stored thereon, wherein the instructions, when executed by at least one processor, cause a device to perform operations comprising:
obtaining an image; determining a portion of the image based on applying a segmentation process to the image; determining a correlation between the portion of the image and each of a plurality of item image templates; determining one of the plurality of item image templates based on the correlations; generating identification data associating the image to an item corresponding to the determined one of the plurality of item image templates; and storing the identification data in a data repository.

20 . The non-transitory computer readable medium of claim 19 , wherein the instructions, when executed by the at least one processor, cause the device to perform operations comprising:
determining an item placement location captured within the image; obtaining planogram data; determining, based on the planogram data, whether the item is assigned to the item placement location.

Assignees

• Walmart Apollo Llc

Inventors

• Karmakar Somedip
• Chakraborty Souradip
• Mishra Abhishek

Classifications

• G06V10/54

  relating to texture · CPC title

• G06T7/73

  using feature-based methods · CPC title

• G06T7/11

  Region-based segmentation · CPC title

• G06V10/443 Primary

  by matching or filtering · CPC title

• G06V20/52

  Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69) · CPC title