Systems and methods for processing images captured at a product storage facility

Patent metadata

Field	Value
Publication number	US-12450883-B2
Application number	US-202318158925-A
Country	US
Kind code	B2
Filing date	Jan 24, 2023
Priority date	Jan 24, 2023
Publication date	Oct 21, 2025
Grant date	Oct 21, 2025

Abstract

In some embodiments, apparatuses and methods are provided herein useful to processing captured images of objects at a product storage facility. In some embodiments, there is provided a system for processing captured images of objects including a trained machine learning model and a control circuit. In some embodiments, the trained machine learning model is configured to process unprocessed captured images. In some embodiments, the control circuit is configured to associate each of the processed images into one of a first group, a second group, or a third group; remove at least one processed image associated with the first group from the processed images in accordance with a first processing rule; and output remaining processed images associated with the first group and processed images associated with the second group to be used to retrain the trained machine learning model.

First claim

What is claimed is:

1. A system for processing captured images of objects at a product storage facility, the system comprising:
a trained machine learning model configured to:
process unprocessed captured images, wherein at least some of the unprocessed captured images depict objects in the product storage facility; and
output processed images; and

a control circuit configured to:
associate each of the processed images into one of a first group, a second group, or a third group,
wherein the first group corresponds to at least one of (a) images depicting one or more objects that are not detected by the trained machine learning model as being associated with a recognized product but a recognized price tag was detected as being associated with the recognized product, or (b) images depicting the one or more objects having at least one of a textual similarity or a visual similarity with a product description stored in a database but the trained machine learning model did not detect as being associated with the recognized product,
wherein the second group corresponds to images depicting one or more objects that are detected by the trained machine learning model as being associated with more than one recognized product, and
wherein the third group corresponds to images depicting one or more objects that the trained machine learning model is unable to detect as depicting an object;

remove the images associated with the third group from the processed images;
calculate a similarity score for each of the processed images in the first group, each similarity score representing the textual similarity or the visual similarity between the processed image and previously processed images stored in the database that are associated with false-negatives;
remove at least one processed image from the first group based on the similarity score for the at least one processed image; and
output remaining processed images associated with the first group and processed images associated with the second group to be used to retrain the trained machine learning model.

2. The system of claim 1 , further comprising:
one or more image capture devices configured to capture images of the objects in the product storage facility; and
the database configured to store at least one of the unprocessed captured images and the processed images.

3. The system of claim 2 , wherein at least one of the one or more image capture devices is coupled to a motorized robotic unit.

4. The system of claim 1 , wherein one or more of the processed images depict at least one or both of an object inside a bounding box and a price tag inside a bounding box.

5. The system of claim 1 , further comprising a user interface coupled to the control circuit, wherein the user interface is configured to be used by a user to at least one of associate a product with at least one depicted object in the remaining processed images associated with the first group or resolve that one or more objects depicted in the images associated with the second group is only associated with a single product, wherein an output of the user interface is used to retrain the trained machine learning model.

6. The system of claim 1 , wherein the unprocessed captured images comprise images that have not gone through object detection or object classification by the control circuit.

7. The system of claim 1 , wherein removing the at least one processed image associated with the first group from the processed images can further include removing processed images that are similar to the previously processed images based on a location similarity with where the previously processed images were captured.

8. The system of claim 1 , wherein others of the unprocessed captured images depict objects at one or more additional product storage facilities.

9. The system of claim 1 , wherein the product storage facility comprises one of a retail store, a distribution center, and a fulfillment center.

10. A method for processing captured images of objects at a product storage facility, the method comprising:
processing, by a trained machine learning model, unprocessed captured images, wherein at least some of the unprocessed captured images depict objects in the product storage facility;
outputting, by the trained machine learning model, processed images;
associating, by a control circuit, each of the processed images into one of a first group, a second group, or a third group,
wherein the first group corresponds to at least one of (a) images depicting one or more objects that are not detected by the trained machine learning model as being associated with a recognized product but a price tag was detected as being associated with the recognized product, or (b) images depicting the one or more objects having at least one of a textual similarity or a visual similarity with a product description stored in a database but the trained machine learning model did not detect as being associated with the recognized product,
wherein the second group to images depicting one or more objects that are detected by the trained machine learning model as being associated with more than one recognized product, and
wherein the third group corresponds to images depicting one or more objects that the trained machine learning model is unable to detect as depicting an object;

removing, by the control circuit, the images associated with the third group from the processed images;
calculating, by the control circuit, a similarity score for each of the processed images in the first group, each similarity score representing the textual similarity or the visual similarity between the processed image and previously processed images stored in the database that are associated with false-negatives;
removing, by the control circuit, at least one processed image from the first group based on the similarity score for the at least one processed image; and
outputting, by the control circuit, remaining processed images associated with the first group and processed images associated with the second group to be used to retrain the trained machine learning model.

11. The method of claim 10 , further comprising:
capturing, by one or more image capture devices, images of the objects in the product storage facility; and
storing, by a database, at least one of the unprocessed captured images and the processed images.

12. The method of claim 11 , wherein at least one of the one or more image capture devices is coupled to a motorized robotic unit.

13. The method of claim 10 , wherein one or more of the processed images depict at least one or both of an object inside a bounding box or a price tag inside a bounding box.

14. The method of claim 10 , further comprising outputting, by a user interface coupled to the control circuit to retrain the trained machine learning model, at least one of an association of a product with at least one depicted object in the remaining processed images associated with the first group or a resolution that one or more objects depicted in the images associated with the second group is only associated with a single product.

15. The method of claim 10 , wherein the unprocessed captured images comprise images that have not gone through objection detection or object classification by the control circuit.

16. The method of claim 10 , wherein removing the at least one processed image associated with the first group from the processed images can further include removing processed images that are similar to the previously processed images based on a location similarity with where the previously processed images were

Assignees

• Walmart Apollo Llc

Inventors

• Balusu Raghava
• Jade Avinash M
• Zhang Lingfeng
• Robinson Jr William C
• Ellison Benjamin R
• Muktevi Srinivas
• Jhunjhunwala Amit
• Duan Zhaoliang
• Chakraborty Siddhartha
• Ghosh Ashlin
• Yuan Mingquan

Classifications

• G06V10/945

  User interactive design; Environments; Toolboxes · CPC title

• G06V10/761

  Proximity, similarity or dissimilarity measures · CPC title

• G06V10/82

  using neural networks · CPC title

• G06V20/50 Primary

  Context or environment of the image · CPC title

• G06V20/52

  Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69) · CPC title