Systems and methods for processing images captured at a product storage facility

Patent metadata

Field	Value
Publication number	US-2024249505-A1
Application number	US-202318158925-A
Country	US
Kind code	A1
Filing date	Jan 24, 2023
Priority date	Jan 24, 2023
Publication date	Jul 25, 2024
Grant date	—

Abstract

In some embodiments, apparatuses and methods are provided herein useful to processing captured images of objects at a product storage facility. In some embodiments, there is provided a system for processing captured images of objects including a trained machine learning model and a control circuit. In some embodiments, the trained machine learning model is configured to process unprocessed captured images. In some embodiments, the control circuit is configured to associate each of the processed images into one of a first group, a second group, or a third group; remove at least one processed image associated with the first group from the processed images in accordance with a first processing rule; and output remaining processed images associated with the first group and processed images associated with the second group to be used to retrain the trained machine learning model.

First claim

What is claimed is:

1 . A system for processing captured images of objects at a product storage facility, the system comprising:
a trained machine learning model configured to:
process unprocessed captured images, wherein at least some of the unprocessed captured images depict objects in the product storage facility; and
output processed images;
a control circuit configured to:
associate each of the processed images into one of a first group, a second group, or a third group,
wherein the first group corresponds to (a) at least one of images depicting one or more objects that are not detected by the trained machine learning model as being associated with a recognized product, (b) the images depicting the one or more objects that are not detected by the trained machine learning model as being associated with the recognized product but a recognized price tag was detected as being associated with the recognized product, or (c) the images depicting the one or more objects having at least one of a textual similarity or a visual similarity with a product description stored in a database but the trained machine learning model did not detect as being associated with the recognized product,
wherein the second group corresponds to images depicting one or more objects that are detected by the trained machine learning model as being associated with more than one recognized product, and
wherein the third group corresponds to images depicting one or more objects that the trained machine learning model is unable to detect as depicting an object;

remove at least one processed image associated with the first group from the processed images in accordance with a first processing rule; and
output remaining processed images associated with the first group and processed images associated with the second group to be used to retrain the trained machine learning model.

2 . The system of claim 1 , further comprising:
one or more image capture devices configured to capture images of the objects in the product storage facility; and the database configured to store at least one of the unprocessed captured images and the processed images.

3 . The system of claim 2 , wherein at least one of the one or more image capture devices is coupled to a motorized robotic unit.

4 . The system of claim 1 , wherein one or more of the processed images depict at least one or both of an object inside a bounding box and a price tag inside a bounding box.

5 . The system of claim 1 , wherein the control circuit is further configured to remove the images associated with the third group from the processed images.

6 . The system of claim 1 , further comprising a user interface coupled to the control circuit, wherein the user interface is configured to be used by a user to at least one of associate a product with at least one depicted object in the remaining processed images associated with the first group or resolve that one or more objects depicted in the images associated with the second group is only associated with a single product, wherein an output of the user interface is used to retrain the trained machine learning model.

7 . The system of claim 1 , wherein the unprocessed captured images comprise images that have not gone through object detection or object classification by the control circuit.

8 . The system of claim 1 , wherein the first processing rule comprises removing images that are similar to previously processed images based on at least one of the textual similarity, the visual similarity, or a location similarity where the previously processed images were captured.

9 . The system of claim 1 , wherein others of the unprocessed captured images depict objects at one or more additional product storage facilities.

10 . The system of claim 1 , wherein the product storage facility comprises one of a retail store, a distribution center, and a fulfillment center.

11 . A method for processing captured images of objects at a product storage facility, the method comprising:
processing, by a trained machine learning model, unprocessed captured images, wherein at least some of the unprocessed captured images depict objects in the product storage facility; outputting, by the trained machine learning model, processed images; associating, by a control circuit, each of the processed images into one of a first group, a second group, or a third group,
wherein the first group corresponds to (a) at least one of images depicting one or more objects that are not detected by the trained machine learning model as being associated with a recognized product, (b) the images depicting the one or more objects that are not detected by the trained machine learning model as being associated with the recognized product but a price tag was detected as being associated with the recognized product, or (c) the images depicting the one or more objects having at least one of a textual similarity or a visual similarity with a product description stored in a database but the trained machine learning model did not detect as being associated with the recognized product,
wherein the second group corresponds to images depicting one or more objects that are detected by the trained machine learning model as being associated with more than one recognized product, and
wherein the third group corresponds to images depicting one or more objects that the trained machine learning model is unable to detect as depicting an object;
removing, by the control circuit, at least one processed image associated with the first group from the processed images in accordance with a first processing rule; outputting, by the control circuit, remaining processed images associated with the first group and processed images associated with the second group to be used to retrain the trained machine learning model.

12 . The method of claim 11 , further comprising:
capturing, by one or more image capture devices, images of the objects in the product storage facility; and storing, by a database, at least one of the unprocessed captured images and the processed images.

13 . The method of claim 12 , wherein at least one of the one or more image capture devices is coupled to a motorized robotic unit.

14 . The method of claim 11 , wherein one or more of the processed images depict at least one or both of an object inside a bounding box or a price tag inside a bounding box.

15 . The method of claim 11 , further comprising removing, by the control circuit, the images associated with the third group from the processed images.

16 . The method of claim 11 , further comprising outputting, by a user interface coupled to the control circuit to retrain the trained machine learning model, at least one of an association of a product with at least one depicted object in the remaining processed images associated with the first group or a resolution that one or more objects depicted in the images associated with the second group is only associated with a single product.

17 . The method of claim 11 , wherein the unprocessed captured images comprise images that have not gone through objection detection or object classification by the control circuit.

18 . The method of claim 11 , wherein the first processing rule comprises removing images that are similar to previously processed images based on at least one of the textual similarity, the visual similarity, or a location similarity where the previously processed images were captured.

19 . The method of claim 11 , wherein others of the unprocessed captured ima

Assignees

• Walmart Apollo Llc

Inventors

• Balusu Raghava
• Jade Avinash M
• Zhang Lingfeng
• Robinson Jr William C
• Ellison Benjamin R
• Muktevi Srinivas
• Jhunjhunwala Amit
• Duan Zhaoliang
• Chakraborty Siddhartha
• Ghosh Ashlin
• Yuan Mingquan

Classifications

• G06V20/52

  Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69) · CPC title

• G06V10/82

  using neural networks · CPC title

• G06V10/761

  Proximity, similarity or dissimilarity measures · CPC title

• G06V10/945

  User interactive design; Environments; Toolboxes · CPC title

• G06V10/774 Primary

  Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting · CPC title