Systems and methods of updating model templates associated with images of retail products at product storage facilities

Patent metadata

Field	Value
Publication number	US-2024257043-A1
Application number	US-202318102999-A
Country	US
Kind code	A1
Filing date	Jan 30, 2023
Priority date	Jan 30, 2023
Publication date	Aug 1, 2024
Grant date	—

Abstract

Systems and methods of updating templates for use in recognizing individual products in images captured at a product storage facility include an image capture device that captures one or more images of product storage structure at a product storage facility, a computing device in communication with the image capture device, and an electronic database that stores keyword model templates and feature model templates associated with images of previously recognized individual products detected at the product storage facility. The computing device obtains the keyword and feature model templates associated with a recognized product from the electronic database, extracts the keywords from the products associated with the obtained keyword model templates, identifies products that are similar to the recognized product, and updates the keyword model template for each of the products to include must keywords and negative keywords, facilitating recognition of products in subsequent images captured by the image capture device.

First claim

What is claimed is:

1 . A system of updating templates for use in recognizing individual products in images captured at a product storage facility, the system comprising:
an image capture device having a field of view that includes at least a portion of a product storage structure of the product storage facility, the product storage structure having the individual products arranged thereon, wherein the image capture device is configured to capture one or more images of the product storage structure; a computing device including a control circuit, the computing device being communicatively coupled to the image capture device; and an electronic database configured to store keyword model templates and feature model templates associated with the images of previously recognized individual products stored at the product storage facility, wherein the keyword model templates includes an image of a recognized individual product and meta data associated with the recognized individual product, and wherein the feature model templates include the image of the recognized product in association with visual features of the recognized product; wherein the control circuit of the computing device is configured to:
obtain at least one of the keyword model templates and feature model templates stored in the electronic database;
extract one or more keywords from each of the individual recognized products depicted in the captured images associated with the obtained keyword model templates;
correlate the keywords extracted from each of the individual recognized products depicted in the images associated with the obtained keyword model templates to identify similar products, where the similar products share a number of keywords with each other and do not share a number of keywords with each other;
update a keyword model template for each of the similar products to:
set the keywords that are unique to the similar recognized products as must keywords; and
set the must keywords that are not shared between the similar recognized products as negative keywords; and

transmit the updated keyword model template including the must keywords and the negative keywords for each of the similar recognized products to the electronic database for storage to be used for analysis of subsequent images captured by the image capture device, and recognition of the products in the subsequent images.

2 . The system of claim 1 , wherein the image capture device comprises a motorized robotic unit that includes wheels that permit the motorized robotic unit to move about the product storage facility, and a camera to permit the motorized robotic unit to capture the one or more images of the product storage structure.

3 . The system of claim 1 , wherein the control circuit sets the keywords that are shared between the similar recognized products as high frequency keywords.

4 . The system of claim 1 , wherein the keyword model templates and the feature model templates are generated based on processing, by the control circuit of the computing device, of a first batch of the images of the product storage structure captured by the image capture device to crop the products detected in the first batch of the images to generate a first batch of cropped images, each of the cropped images of the first batch depicting an individual one of the products detected in the first batch of images.

5 . The system of claim 4 , wherein the control circuit is programmed to:
obtain a second batch of images of the product structure captured by the image capture device; analyze the obtained images of the second batch to detect individual ones of the products located on the product storage structure; based on detection of the individual ones of the products in the images of the second batch, recognize the individual ones of the products detected in the images of the second batch as corresponding to a known product identifier; crop each of the individual ones of the recognized products from the images of the second batch to generate a second batch of cropped images; combine the cropped images of the first batch that correspond to the known product identifier and the cropped images of the second batch to create a plurality of image clusters, wherein each of the image clusters contains the cropped images that depict an identical one of the recognized individual products corresponding to the known product identifier; and select one of the cropped images in each of the image clusters as a centroid image, wherein the centroid image is designated as the updated keyword model template representing the one of the recognized individual products.

6 . The system of claim 5 , wherein the control circuit is programmed to:
generate embeddings for each of the cropped images of the second batch, wherein the embeddings represent dense vector representations of the cropped images of the second batch; determine a similarly of the embeddings between the cropped images of the second batch in each of the image clusters; and position the cropped images of the second batch in each of the image clusters based on a similarity between the embeddings of the cropped images of the second batch in each of the clusters.

7 . The system of claim 6 , wherein, after selection of the centroid image of an image cluster, the control circuit is programmed to:
resample a number of the cropped images of the cluster that are located closest to the centroid image by virtue of having embeddings that are most similar to the embeddings of the centroid image; and select the centroid image and the resampled images as an updated feature model template representing the one of the recognized individual products.

8 . The system of claim 7 ,
wherein the control circuit is programmed to perform optical character recognition to detect one or more keywords from each of the individual products depicted in the cropped images of the second batch; and wherein the updated keyword model template includes additional keywords detected via the optical character recognition in at least one of the cropped images of the second batch.

9 . The system of claim 7 ,
wherein the control circuit is programmed to correlate embeddings of the cropped images of the first batch used to generate the keyword model templates and the feature model templates stored in the electronic database with the embeddings generated by the control circuit for the cropped images of the second batch; and wherein the updated feature model template includes at least some of the embeddings associated with the cropped images of the second batch.

10 . The system of claim 7 ,
wherein the control circuit is programmed to remove at least one of the cropped images of the first batch from the feature model template previously stored in the electronic database based on a determination that the at least one of the cropped images of the first batch was incorrectly associated with the feature model template previously stored in the electronic database; and wherein the control circuit is programmed to remove at least one of the cropped images of the second batch from the updated feature model template based on a determination that the at least one of the cropped images of the second batch of images was incorrectly associated with the updated feature model template.

11 . A method of updating templates for use in recognizing individual products in images captured at a product storage facility, the method comprising:
capturing one or more images of a product structure of the product storage facility by an image capture device having a field of view that includes at least a portion of the product storage structure, the product storage structu

Assignees

• Walmart Apollo Llc

Inventors

• Zhang Han
• Pachauri Abhinav
• Balusu Raghava
• Ghosh Ashlin
• Jade Avinash M
• Zhang Lingfeng
• Muktevi Srinivas
• Jhunjhunwala Amit
• Duan Zhaoliang

Classifications

• G06Q10/087 Primary

  Inventory or stock management, e.g. order filling, procurement or balancing against orders · CPC title

• G06V20/60 Primary

  Type of objects · CPC title

• G06V2201/07

  Target detection · CPC title