Methods and systems for creating reference image templates for identification of products on product storage structures of a product storage facility

Patent metadata

Field	Value
Publication number	US-2024249239-A1
Application number	US-202318158983-A
Country	US
Kind code	A1
Filing date	Jan 24, 2023
Priority date	Jan 24, 2023
Publication date	Jul 25, 2024
Grant date	—

Abstract

Systems and methods of creating reference template images for detecting and recognizing products at a product storage facility include an image capture device having a field of view that includes a product storage structure of the product storage facility, and a computing device including a control circuit and being communicatively coupled to the image capture device. The computing device obtains images of the product storage structure captured by the image capture device, analyzes the obtained images to detect individual ones of the products located on the product storage structure. Then, the computing device identifies the individual ones of the products detected in the images and crops each of the individual ones of the identified products from the images to generate cropped images. The computing device then creates a cluster of the cropped images, and selects one of the cropped images as a reference template image of an identified individual product.

First claim

What is claimed is:

1 . A system of creating reference template images for detecting and recognizing products at product storage areas of a product storage facility, the system comprising:
an image capture device having a field of view that includes at least a portion of a product storage structure in a product storage area of the product storage facility, the product storage structure having products arranged thereon, wherein the image capture device is configured to capture one or more images of the product storage structure; and a computing device including a control circuit, the computing device being communicatively coupled to the image capture device, the control circuit being configured to:
obtain a plurality of images of the product storage structure captured by the image capture device;
analyze the obtained images of the product storage structure captured by the image capture device to detect individual ones of the products located on the product storage structure;
based on detection of the individual ones of the products in the images, recognize the individual ones of the products detected in the images as corresponding to a known product identifier;
crop each of the individual ones of the recognized products from the images to generate a plurality of cropped images;
create a cluster of the cropped images, wherein each of the cropped images in the cluster depicts one of the recognized individual products; and
analyze the cluster of the cropped images to select one of the cropped images as a reference template image representing the one of the recognized individual products.

2 . The system of claim 1 , wherein the image capture device comprises a motorized robotic unit that includes wheels that permit the motorized robotic unit to move about the product storage facility, and a camera to permit the motorized robotic unit to capture the one or more images of the product storage structure.

3 . The system of claim 1 , wherein the control circuit is programmed to generate virtual boundary lines each of the obtained images, wherein each of the virtual boundary lines surrounds an individual one of the products captured in the obtained images.

4 . The system of claim 1 , wherein the control circuit is programmed to generate embeddings for each of the cropped images, wherein the embeddings represent dense vector representations of the images.

5 . The system of claim 4 , wherein the control circuit is programmed to generate the embeddings for each of the cropped images using a convolutional neural network pretrained to extract predetermined features from the cropped images and to generate a lower dimensional representation of the cropped images.

6 . The system of claim 5 , wherein the control circuit is programmed to:
group the cropped images containing the embeddings into the cluster, wherein each of the cropped images in the cluster depicts one of the recognized individual products; and select a centroid image of the cluster of the cropped images, wherein the centroid image is determined by the control circuit to represent a keyword template reference image of the one of the recognized individual products.

7 . The system of claim 6 , wherein the control circuit is programmed to:
determine a similarly of the embeddings between the cropped images in the cluster; and position the cropped images in the cluster based on a similarity between the embeddings of the cropped images in the cluster.

8 . The system of claim 6 , wherein, after selection of the centroid image of the cluster of the cropped images, the control circuit is programmed to:
resample a predetermined number of images of the cluster of the cropped images that are located closest to the centroid image; and mark the centroid and the resampled images as feature vector template reference images of the one of the recognized individual products.

9 . The system of claim 8 , further comprising an electronic database that stores the keyword template reference images and the feature vector template reference images associated with each one of the recognized individual products to facilitate recognition of the products subsequently captured in at least one new image of the product storage structure by the image capture device.

10 . The system of claim 9 , wherein the control circuit is programmed to replace the one of the cropped images as the keyword template reference images or the feature vector template reference images of the one of the recognized individual products in response to a determination by the control circuit that another cropped image obtained from the at least one new image represents the centroid in an updated cluster of the cropped images of the one of the recognized individual products.

11 . A method of creating reference template images for detecting and recognizing products at product storage areas of a product storage facility, the method comprising:
capturing one or more images of a product storage structure in a product storage area of the product storage facility via an image capture device having a field of view that includes the product storage structure, the product storage structure having products arranged thereon; and by a computing device including a control circuit and communicatively coupled to the image capture device:
obtaining a plurality of images of the product storage structure captured by the image capture device;
analyzing the obtained images of the product storage structure captured by the image capture device to detect individual ones of the products located on the product storage structure;
based on detection of the individual ones of the products in the images, recognizing the individual ones of the products detected in the images as corresponding to a known product identifier;
cropping each of the individual ones of the recognized products from the images to generate a plurality of cropped images;
creating a cluster of the cropped images, wherein each of the cropped images in the cluster depicts one of the recognized individual products; and
analyzing the cluster of the cropped images to select one of the cropped images as a reference template image representing the one of the recognized individual products.

12 . The method of claim 11 , wherein the image capture device comprises a motorized robotic unit that includes wheels that permit the motorized robotic unit to move about the product storage facility, and a camera to permit the motorized robotic unit to capture the one or more images of the product storage structure.

13 . The method of claim 11 , further comprising, by the control circuit, generating virtual boundary lines each of the obtained images, wherein each of the virtual boundary lines surrounds an individual one of the products captured in the obtained images.

14 . The method of claim 11 , further comprising, by the control circuit, generating embeddings for each of the cropped images, wherein the embeddings represent dense vector representations of the images.

15 . The method of claim 14 , further comprising, by the control circuit, generating the embeddings for each of the cropped images using a convolutional neural network pretrained to extract predetermined features from the cropped images and to generate a lower dimensional representation of the cropped images.

16 . The method of claim 15 , further comprising, by the control circuit:
grouping the cropped images containing the embeddings into the cluster, wherein each of the cropped images in the cluster depicts one of the recognized individual products; and selecting a centroi

Assignees

• Walmart Apollo Llc

Inventors

• Ghosh Ashlin
• Balusu Raghava
• Pachauri Abhinav
• Jade Avinash M
• Zhang Lingfeng
• Jhunjhunwala Amit
• Robinson Jr William Craig
• Ellison Benjamin R
• Muktevi Srinivas
• Duan Zhaoliang

Classifications

• G06Q10/087 Primary

  Inventory or stock management, e.g. order filling, procurement or balancing against orders · CPC title