Systems and methods for analyzing depth in images obtained in product storage facilities to detect outlier items

Patent metadata

Field	Value
Publication number	US-12567167-B2
Application number	US-202318165152-A
Country	US
Kind code	B2
Filing date	Feb 6, 2023
Priority date	Feb 6, 2023
Publication date	Mar 3, 2026
Grant date	Mar 3, 2026

Abstract

In some embodiments, apparatuses and methods are provided herein useful to processing captured images. In some embodiments, there is provided a system for processing captured images of objects at a product storage facility including a trained machine learning model stored in a memory; and a control circuit. The control circuit may obtain an image at the product storage facility; cluster objects depicted in the image that have same product identifiers into a corresponding group; determine coordinates of each bounding box of each clustered object in the corresponding group; determine a bounding box representative depth value of pixels inside the bounding box of each clustered object; determine an overall representative depth value of the corresponding group based on bounding box representative depth values of clustered objects; and exclude the clustered objects from identified objects in the image upon a determination that the overall representative depth value is greater than a threshold.

First claim

What is claimed is:

1 . A system comprising:
at least one of an autonomous floor cleaner or an autonomous floor sweeper configured to clean or sweep a product storage facility; at least one image capture device incorporated into the at least one autonomous floor cleaner or autonomous floor sweeper, the at least one image capture device configured to capture an image of objects associated with a product storage structure at the product storage facility as the at least one autonomous floor cleaner or autonomous floor sweeper moves throughout the product storage facility; a trained machine learning model stored in a memory; and a control circuit executing the trained machine learning model, the control circuit configured to: obtain the image of the objects associated with the product storage structure; cluster a subset of the objects having a common product identifier into a group of clustered objects, wherein clustering the subset of objects into the group of clustered objects includes determining a bounding box that encircles the group of clustered objects; determine a bounding box representative depth value of pixels inside the bounding box; determine an overall representative depth value for the group of clustered objects based on the bounding box representative depth value; exclude the group of clustered objects from tracked inventory items housed by the product storage structure in response to determining that the overall representative depth value is greater than a threshold, the tracked inventory items housed by the product storage structure being used for inventory management at the product storage facility, wherein the group of clustered objects are background objects positioned behind the product storage structure such that the group of clustered objects are not housed by the product storage structure; and send an alert instructing a worker to inspect the product storage to confirm that the clustered objects are background objects.

2 . The system of claim 1 , wherein the objects comprise items for commercial sale.

3 . The system of claim 1 , wherein the bounding box representative depth value is determined based on one or both of an average depth value and a median depth value of the pixels inside the bounding box.

4 . The system of claim 1 , wherein the group of clustered objects include background objects housed by a different product storage structure within the product storage facility.

5 . The system of claim 1 , wherein the product storage structure is a shelf, a pallet, a bin, or a rack in the product storage facility.

6 . The system of claim 1 , wherein the threshold is determined based on a lowest depth value associated with one or more of the objects associated with the product storage structure, the one or more objects being omitted from the group of clustered objects.

7 . The system of claim 6 , wherein the threshold is determined by adding a constant value to the lowest depth value.

8 . A method comprising:
capturing, by at least one image capture device incorporated into at least one of an autonomous floor cleaner or an autonomous floor sweeper, an image of objects associated with a product storage structure at a product storage facility as the at least one autonomous floor cleaner or autonomous floor sweeper moves throughout the product storage facility, wherein the at least one autonomous floor cleaner or autonomous floor sweeper is configured to clean or sweep the product storage facility; obtaining, by a control circuit executing a trained machine learning model stored in a memory, the image of the objects associated with the product storage structure; clustering, by the control circuit executing the trained machine learning model, a subset of the objects having a common product identifier into a group of clustered objects, wherein clustering the subset of objects into the group of clustered objects includes determining a bounding box that encircles the group of clustered objects; determining, by the control circuit executing the trained machine learning model, a bounding box representative depth value of pixels inside the bounding box; determining, by the control circuit executing the trained machine learning model, an overall representative depth value for the group of clustered objects based on the bounding box representative depth value; excluding, by the control circuit executing the trained machine learning model, the group of clustered objects from tracked inventory items housed by the product storage structure in response to determining that the overall representative depth value is greater than a threshold, the tracked inventory items housed by the product storage structure being used for inventory management at the product storage facility, wherein the group of clustered objects are background objects positioned behind the product storage structure such that the group of clustered objects are not housed by the product storage structure; and sending, by the control circuit, an alert instructing a worker to inspect the product storage to confirm that the clustered objects are background objects.

9 . The method of claim 8 , wherein the objects comprise items for commercial sale.

10 . The method of claim 8 , wherein the bounding box representative depth value is determined based on one or both an average depth value or a median depth value of the pixels inside the bounding box.

11 . The method of claim 8 , wherein the group of clustered objects include background objects housed by a different product storage structure within the product storage facility.

12 . The method of claim 8 , wherein the product storage structure is a shelf, a pallet, a bin, or a rack in the product storage facility.

13 . The method of claim 8 , wherein the threshold is determined based on a lowest depth value associated with one or more of the objects associated with the product storage structure, the one or more objects being omitted from the group of clustered objects.

14 . The method of claim 13 , wherein the threshold is determined by adding a constant value to the lowest depth value.

15 . A system comprising:
at least one of an autonomous floor cleaner or an autonomous floor sweeper configured to clean or sweep a product storage facility; at least one image capture device incorporated into the at least one autonomous floor cleaner or autonomous floor sweeper, the at least one image capture device configured to capture an image of objects associated with a product storage structure at the product storage facility as the at least one autonomous floor cleaner or autonomous floor sweeper moves throughout the product storage facility; and one or more processors configured to execute a trained machine learning model to perform the following operations:
clustering a subset of the objects having a first product identifier into a group of clustered objects, the group of clustered objects encircled by a bounding box;
determining that a depth value associated with the bounding box is greater than a threshold;
excluding the group of clustered objects from tracked inventory items housed by the product storage structure in response to the depth value associated with the bounding box being greater than a threshold, the tracked inventory items housed by the product storage structure being used for inventory management at the product storage facility, wherein the group of clustered objects are background objects positioned behind the product storage structure such that the group of clustered objects are not housed by the product storage structure; and
sending an alert instructi

Assignees

• Walmart Apollo Llc

Inventors

• Zhang Han
• Chen Yilun
• Zhang Lingfeng
• Cantor Adam
• Jade Avinash M
• Ellison Benjamin R
• Robinson Jr William Craig
• Yuan Mingquan
• Duan Zhaoliang
• Wang Wei

Classifications

• G06V10/25 Primary

  Determination of region of interest [ROI] or a volume of interest [VOI] · CPC title

• G06V10/762

  using clustering, e.g. of similar faces in social networks · CPC title

• G06V2201/07

  Target detection · CPC title

• G06T7/50

  Depth or shape recovery · CPC title

• G06T7/70 Primary

  Determining position or orientation of objects or cameras (camera calibration G06T7/80) · CPC title