Systems and methods for analyzing and labeling images in a retail facility

Patent metadata

Field	Value
Publication number	US-2024257047-A1
Application number	US-202318161788-A
Country	US
Kind code	A1
Filing date	Jan 30, 2023
Priority date	Jan 30, 2023
Publication date	Aug 1, 2024
Grant date	—

Abstract

In some embodiments, apparatuses and methods are provided herein useful to processing captured images. In some embodiments, there is provided a system for processing captured images of objects including a memory and a control circuit executing a trained machine learning model. The memory may be configured to store a plurality of images comprising first images and second images. The control circuit may be configured to: allocate each of the first images into one of a plurality of datasets; cluster each image in the dataset into one of a plurality of groups; select a sample from at least one of the plurality of groups; cluster each of the second images into one of dominant product identifier group and a non-dominant product identifier group; select a sample from the dominant product identifier group and a sample from the non-dominant product identifier group; and output the selected sample.

First claim

What is claimed is:

1 . A system for processing captured images of objects at a product storage facility, the system comprising:
a memory configured to store a plurality of images, the plurality of images comprising first images and second images, wherein each of the first images contain items not detected by a trained machine learning model as being associated with a recognized product identifier, and wherein each of the second images contain items detected by the trained machine learning model as being associated with multiple recognized product identifiers; and a control circuit executing the trained machine learning model configured to:
allocate each of the first images into one of a plurality of datasets based on areas in the product storage facility the first images were captured;
for each dataset of the plurality of datasets, cluster each image in the dataset into one of a plurality of groups based on a degree of resemblance of items depicted in the image relative to those items depicted in other images in the dataset;
select a sample from at least one of the plurality of groups;
cluster each of the second images into one of dominant product identifier group and a non-dominant product identifier group;
select a sample from the dominant product identifier group;
select a sample from the non-dominant product identifier group; and
output at least one of the selected sample from the at least one of the plurality of groups, the selected sample from the dominant product identifier group, and the selected sample from the non-dominant product identifier group to be used to retrain the trained machine learning model.

2 . The system of claim 1 , wherein the plurality of groups comprise a homogenous group, a heterogeneous group, a low similarity group, and an individual group.

3 . The system of claim 2 , wherein the homogenous group includes images where the degree of resemblance of items depicted in each image is greater than 70%.

4 . The system of claim 2 , wherein the heterogeneous group includes images where the degree of resemblance of items depicted in each image is at least 30% but no more than 70%.

5 . The system of claim 2 , wherein the low similarity group includes images where the degree of resemblance of items depicted in each image is less than 30%.

6 . The system of claim 2 , wherein the individual group includes images that do not have resemblance with any other image in the dataset.

7 . The system of claim 2 , wherein the control circuit executing the trained machine learning model is further configured to:
process each image included in the low similarity group by determining the degree of resemblance of items depicted in the image, and clustering the image into one of the homogenous group, the heterogeneous group, the low similarity group, or the individual group in accordance with the determined degree of resemblance of the items; and repeat processing of subsequent images included in the low similarity group until each of the subsequent images is clustered into one of the homogenous group, the heterogeneous group, or the individual group in accordance with the determined degree of resemblance of the items.

8 . The system of claim 1 , wherein the control circuit executing the trained machine learning model is further configured to determine the degree of resemblance of items based on at least one of a textual similarity or a visual similarity of the items.

9 . The system of claim 1 , wherein the selected sample from the at least one of the plurality of groups comprises a sample of one or more images based on a rule from each of the plurality of groups.

10 . The system of claim 9 , wherein the control circuit executing the trained machine learning model is further configured to group the sample of the one or more images from each of the plurality of groups into the selected sample, wherein the selected sample is output in response to a determination that an image count of the selected sample is less than a threshold.

11 . The system of claim 10 , wherein the control circuit executing the trained machine learning model is further configured to:
in response to a determination that the image count of the selected sample is at least the threshold, allocate each image in the selected sample into one of the plurality of datasets based on the areas in the product storage facility images in the selected sample were captured; for each dataset of the plurality of datasets, cluster each image in the dataset into one of the plurality of groups based on the degree of resemblance of items depicted in the image relative to those items depicted in other images in the dataset; select a second sample from at least one of the plurality of groups; and determine whether a second image count of the second sample is less than the threshold, wherein the allocation of each image in the selected sample into one of the plurality of datasets, the clustering of each image in the dataset into one of the plurality of groups, and the selection of a subsequent sample from at least one of the plurality of groups are repeated until the subsequent sample is less than the threshold.

12 . The system of claim 9 , wherein the plurality of groups comprise a homogenous group, a heterogeneous group, a low similarity group, and an individual group, and wherein the rule comprises:
selecting, by the control circuit executing the trained machine learning model and for images included in the homogenous group, at least two or three images based on a highest number of at least one of depicted bounding boxes, boundary box aspect ratio, or Optical Character Recognition (OCR) count; selecting, by the control circuit executing the trained machine learning model and for images included in the heterogeneous group, a predetermined percentage of the images included in the heterogeneous group; and selecting, by the control circuit executing the trained machine learning model and for images included in the individual group, all images included in the individual group.

13 . The system of claim 1 , wherein the selected sample from the dominant product identifier group comprises an image having a threshold range of depicted items being associated with a dominant product identifier, wherein the dominant product identifier is a product identifier that is most identified with the depicted items relative to other identified product identifiers of the depicted items in the image.

14 . The system of claim 1 , wherein the selected sample from the non-dominant product identifier group comprises all images associated with the non-dominant product identifier group.

15 . The system of claim 1 , wherein the areas comprise one of an aisle, a bin, a pallet, or a rack storing one or more office supply products, grocery products, electronic products, and household supply products.

16 . A method for processing captured images of objects at a product storage facility, the method comprising:
storing, by a memory, a plurality of images, the plurality of images comprising first images and second images, wherein each of the first images contain items not detected by a trained machine learning model as being associated with a recognized product identifier, and wherein each of the second images contain items detected by the trained machine learning model as being associated with multiple recognized product identifiers; and allocating, by a control circuit executing the trained machine learning model, each of the first images into one of a plurality of datasets based on areas in the product storage facility the first images

Assignees

• Walmart Apollo Llc

Inventors

• Balusu Raghava
• Chakraborty Siddhartha
• Ghosh Ashlin
• Jade Avinash M
• Zhang Lingfeng
• Jhunjhunwala Amit

Classifications

• G06V20/52

  Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69) · CPC title

• G06Q10/087 Primary

  Inventory or stock management, e.g. order filling, procurement or balancing against orders · CPC title

• G06V10/762 Primary

  using clustering, e.g. of similar faces in social networks · CPC title