Systems and methods of detecting price tags and associating the price tags with products

Patent metadata

Field	Value
Publication number	US-12333488-B2
Application number	US-202217971350-A
Country	US
Kind code	B2
Filing date	Oct 21, 2022
Priority date	Oct 21, 2022
Publication date	Jun 17, 2025
Grant date	Jun 17, 2025

Abstract

Systems and methods of analyzing on-shelf price tag labels and products at a product storage facility include an image capture device that captures one or more images of one or more product storage structures at a product storage facility. A computing device communicatively coupled to the image capture device analyzes the images of the product storage structures captured by the image capture device and detects individual price tag labels and individual products located on the product storage structure. Based on the detection of the price tag labels and the products, the computing device also defines separate product storage spaces of the product storage structure, determines which price tag labels are allocated to which of the separate product storage spaces, and associates in a database the price tag labels allocated to the product storage spaces with the products stored in those product storage spaces.

First claim

What is claimed is:

1. A system for analyzing on-shelf price tag labels and products at a product storage facility, the system comprising:
a motorized robotic unit, wherein the motorized robotic unit includes wheels permitting the motorized robotic unit to move about the product storage facility;
an image capture device, implemented on the motorized robotic unit, having a field of view that includes at least a portion of a product storage structure at the product storage facility having the products arranged thereon, wherein the image capture device is configured to capture one or more images from one or more angles of at least the portion of the product storage structure;
one or more computer vision models communicatively coupled to a control circuit of a computing device, wherein the one or more computer vision models are trained using data sets including at least raw image data collected from the captured one or more images, reference image data associated with reference images of one or more product storage structures, reference images of one or more products, reference images of one or more price tag labels, and planogram data associated with the product storage facility; and
the computing device including the control circuit, the computing device being communicatively coupled to the image capture device, the control circuit being configured to use the one or more computer vision models to:
analyze the at least one image of the one or more images of at least the portion of the product storage structure captured by the image capture device to detect individual ones of one or more price tag labels located on the product storage structure within the at least one image;
analyze the at least one image of at least the portion of the product storage structure to detect individual ones of the one or more products located on the product storage structure;
based on the detection of the individual ones of the one more on-shelf products located on at least the portion of the product storage structure, define one or more separate product storage spaces of the product storage structure, wherein each of the one or more separate product storage spaces contains a group of one or more identical products therein;
based on detection of the individual ones of the one or more on-shelf price tag labels located on the product storage structure and based on a definition for each of the one or more separate product storage spaces of the product storage structure, determine at least a first price tag label that is allocated to at least a first product storage space of the separate product storage spaces of the product storage structure; and
based on a determination of at least the first price tag label that is allocated to at least the first product storage space, associate at least the first price tag label with each of the one or more products in the group of the one or more identical products stored in the first product storage space.

2. The system of claim 1 , wherein the control circuit is programmed to generate a first set of virtual boundary lines and a second set of virtual boundary lines, wherein each of the virtual boundary lines of the first set surrounds an individual one of the price tag labels detected in the at least one image, and wherein each of the virtual boundary lines of the second set surrounds an individual one of the products detected in the at least one image.

3. The system of claim 2 ,
wherein the product storage structure comprises a plurality of horizontal support members interconnected by vertical support members; and
wherein the control circuit is further programmed to:
analyze the at least one image of the product storage structure to detect individual ones of the vertical support members of the product storage structure; and
based on detection of the individual ones of the vertical support members of the product storage structure, generate a third set of virtual boundary lines, wherein each one of the virtual boundary lines of the third set surrounds an individual one of the vertical support members detected in the at least one image.

4. The system of claim 3 , wherein the virtual boundary lines of the first set, the second set, and the third set are 2-dimensional, and wherein the control circuit is further programmed to:
estimate depth from the horizontal support members of the product storage structure to the image capture device based on actual size of the price tag labels in the at least one image and based on a pixel size of the price tag labels in the at least one image;
project the virtual boundary lines of the first set, the second set, and the third set into a 3-dimensional coordinate system based on the estimated depth;
align points of the 3-dimensional coordinate system based on location information received from the at least one image capture device; and
re-project the aligned points of the 3-dimensional coordinate system back to a 2-dimensional space.

5. The system of claim 4 , wherein the control circuit is further programmed to cluster the virtual boundary lines of at least one of the first, second, and third sets to define each of the separate product storage spaces containing the group of the identical products therein.

6. The system of claim 5 ,
wherein the control circuit is programmed to generate a fourth set of virtual boundary lines, wherein each of the virtual boundary lines of the fourth set surrounds one of the defined separate product storage spaces containing the group of the identical products therein; and
wherein the first product storage space has two or more adjacent separate product storage spaces that are each surrounded by virtual boundary lines of the fourth set that adjoin the virtual boundary lines of the fourth set that surround the first product storage space.

7. The system of claim 5 , wherein the control circuit is programmed to define each of the separate product storage spaces containing the group of the identical products therein based at least on one or both of center cluster coordinates of the price tag labels and center cluster coordinates of the vertical support members.

8. The system of claim 1 , wherein, based on a determination by the control circuit that at least one of the separate product storage spaces does not have a price tag label allocated thereto, the control circuit is programmed to generate a missing label alert and to transmit the missing label alert.

9. The system of claim 1 , further comprising an electronic database configured to store the images captured by the image capture device, and wherein the control circuit is programmed to transmit, to the electronic database for storage, a signal including electronic data associating the first price tag label with each of the products in the group of identical products stored in the first product storage space.

10. A method of analyzing on-shelf price tag labels and products at a product storage facility, the method comprising:
capturing, via an image capture device having a field of view that includes at least a portion of a product storage structure implemented on a motorized robotic unit, one or more images of a product storage structure at the product storage facility having the products arranged thereon, wherein the motorized robotic unit includes wheels permitting the motorized robotic unit to move about the product storage facility;
obtaining, by a computing device communicatively coupled to the image capture device and including a control circuit, at least one image of the product storage structure captured by the image capture device;
by the control circuit of the computing device:
analyzing, using one or more computer vision models communicatively coupled to a control circ

Assignees

• Walmart Apollo Llc

Inventors

• Wang Jing
• Zhang Han
• Zhang Lingfeng
• Duan Zhaoliang
• Yuan Mingquan
• Wang Wei
• Ellison Benjamin R
• Jade Avinash M
• Balusu Raghava
• Xiao Zhichun

Classifications

• G06T7/62

  of area, perimeter, diameter or volume · CPC title

• G06V10/762

  using clustering, e.g. of similar faces in social networks · CPC title

• G06V10/25

  Determination of region of interest [ROI] or a volume of interest [VOI] · CPC title

• G06V30/41

  Analysis of document content (recognition of printed characters based on code marks G06V30/224) · CPC title

• G06T7/50

  Depth or shape recovery · CPC title