Systems and methods for recognizing product labels and products located on product storage structures of product storage facilities

Patent metadata

Field	Value
Publication number	US-2024265663-A1
Application number	US-202318106269-A
Country	US
Kind code	A1
Filing date	Feb 6, 2023
Priority date	Feb 6, 2023
Publication date	Aug 8, 2024
Grant date	—

Abstract

Systems and methods of pairing product labels with products located on a product storage structure of a product storage facility include an image capture device that captures one or more images of the product storage structure and a computing device that obtains images of the product storage structure captured by the image capture device, analyzes the obtained images to detect product labels and products located on the product storage structure, and crops the detected individual products and individual price tag labels from the images to generate cropped images. Then the computing device stitches the cropped price tag label and product images, receives one or more characters extracted from the portions of the stitched images corresponding to the cropped images, and associates, based on known positional coordinates of the products and product labels in the stitched images, the received extracted characters with corresponding cropped images of the products and product labels.

First claim

What is claimed is:

1 . A system for use in processing images of product labels and products located on a product storage structure of a product storage facility, the system comprising:
an image capture device having a field of view that includes at least a portion of the product storage structure and being configured to capture one or more images of the product storage structure; and a computing device including a control circuit, the computing device being communicatively coupled to the image capture device, the control circuit being configured to:
obtain at least one image of the product storage structure captured by the image capture device;
analyze the at least one image of the product storage structure captured by the image capture device to detect at least one of individual ones of product labels and products located on the product storage structure;
crop each one of the detected individual products and each one of the detected individual product labels from the at least one image to generate a plurality of cropped images, each of the cropped images depicting an individual one of the detected products or an individual one of the detected product labels;
stitch together two or more of the cropped images to generate at least one stitched image;
receive one or more characters extracted from each of the products and each of the product labels detected in the at least one stitched image; and
associate, based on known positional coordinates of each of the products and each of the product labels in the at least one stitched image, the received one or more characters extracted from each one of the individual products and product labels detected in the at least one stitched image with corresponding ones of the plurality of cropped images of the products and product labels.

2 . The system of claim 1 , wherein the image capture device comprises a motorized robotic unit that includes wheels that permit the motorized robotic unit to move about the product storage facility, and a camera to permit the motorized robotic unit to capture the one or more images of the product storage structure.

3 . The system of claim 1 ,
wherein the control circuit is programmed to generate a first set of virtual boundary lines in the at least one image, wherein each of the virtual boundary lines of the first set surrounds an individual one of the products detected in the at least one image; and wherein the control circuit is programmed to generate a second set of virtual boundary lines in the at least one image, wherein each of the virtual boundary lines of the second set surrounds an individual one of the product labels detected in the at least one image.

4 . The system of claim 3 , wherein the at least one stitched image has a predetermined pixel size, and wherein the control circuit is programmed to determine a pixel size of each of the cropped images and to employ a first fit decreasing height algorithm to maximally populate the at least one stitched image with at least one of the cropped images.

5 . The system of claim 4 , wherein the control circuit is programmed to stitch at least two of the cropped images of the individual ones of the detected products and product labels to generate the at least one stitched image by implementing a synchronous architecture in combination with the first fit decreasing height algorithm.

6 . The system of claim 4 , wherein the control circuit is programmed to stitch at least two of the cropped images of the individual ones of the detected products and product labels to generate the at least one stitched image by implementing an asynchronous architecture in combination with the first fit decreasing height algorithm.

7 . The system of claim 1 , wherein the control circuit is programmed to assign a positional coordinate to each of the cropped images of at least one of the products and product labels populated into the at least one stitched image.

8 . The system of claim 7 , wherein the assigned positional coordinate is defined by x and y pixel ranges of the at least one stitched image containing each of the plurality of cropped images.

9 . The system of claim 1 , wherein the control circuit is programmed to receive the one or more characters extracted from each of the products and each of the product labels detected in the at least one stitched image together with positional coordinates of the one more characters within the at least one stitched image.

10 . The system of claim 1 , wherein, after the control circuit associates the received one or more characters extracted from each one of the individual products and product labels detected in the at least one stitched image with respectively corresponding ones of the plurality of cropped images of the products and product labels, the control circuit is programmed to make a prediction of which known product identifiers of the products stocked at the product storage facility present a match to the plurality of cropped images of the products and product labels.

11 . A method of processing images of product labels and products located on a product storage structure of a product storage facility, the method comprising:
capturing one or more images of the product storage structure with an image capture device having a field of view that includes at least a portion of the product storage structure; and by a computing device including a control circuit and being communicatively coupled to the image capture device:
obtaining at least one image of the product storage structure captured by the image capture device;
analyzing the at least one image of the product storage structure captured by the image capture device to detect at least one of individual ones of product labels and products located on the product storage structure;
cropping each one of the detected individual products and each one of the detected individual product labels from the at least one image to generate a plurality of cropped images, each of the cropped images depicting an individual one of the detected products or an individual one of the detected product labels;
stitching together two or more of the cropped images to generate at least one stitched image;
receiving one or more characters extracted from each of the products and each of the product labels detected in the at least one stitched image; and
associating, based on known positional coordinates of each of the products and each of the product labels in the at least one stitched image, the received one or more characters extracted from each one of the individual products and product labels detected in the at least one stitched image with corresponding ones of the plurality of cropped images of the products and product labels.

12 . The method of claim 11 , wherein the image capture device comprises a motorized robotic unit that includes wheels that permit the motorized robotic unit to move about the product storage facility, and a camera to permit the motorized robotic unit to capture the one or more images of the product storage structure.

13 . The method of claim 11 , further comprising, by the control circuit:
generating a first set of virtual boundary lines in the at least one image, wherein each of the virtual boundary lines of the first set surrounds an individual one of the products detected in the at least one image; and generating a second set of virtual boundary lines in the at least one image, wherein each of the virtual boundary lines of the second set surrounds an individual one of the product labels detected in the at least one image.

14 . The method of claim 13 , wherein the at least one stitched image has

Assignees

• Walmart Apollo Llc

Inventors

• Duan Zhaoliang
• Yuan Mingquan
• Lobo Paul Lewis
• Zhang Lingfeng
• Jade Avinash M
• Tang Yutao

Classifications

• G06T7/70

  Determining position or orientation of objects or cameras (camera calibration G06T7/80) · CPC title

• G06Q10/087

  Inventory or stock management, e.g. order filling, procurement or balancing against orders · CPC title

• G06V10/16 Primary

  using multiple overlapping images; Image stitching · CPC title

• G06V30/10

  Character recognition · CPC title

• G06V10/25

  Determination of region of interest [ROI] or a volume of interest [VOI] · CPC title