Systems and methods for analyzing depth in images obtained in product storage facilities to detect outlier items

Patent metadata

Field	Value
Publication number	US-2024265565-A1
Application number	US-202318165152-A
Country	US
Kind code	A1
Filing date	Feb 6, 2023
Priority date	Feb 6, 2023
Publication date	Aug 8, 2024
Grant date	—

Abstract

In some embodiments, apparatuses and methods are provided herein useful to processing captured images. In some embodiments, there is provided a system for processing captured images of objects at a product storage facility including a trained machine learning model stored in a memory; and a control circuit. The control circuit may obtain an image at the product storage facility; cluster objects depicted in the image that have same product identifiers into a corresponding group; determine coordinates of each bounding box of each clustered object in the corresponding group; determine a bounding box representative depth value of pixels inside the bounding box of each clustered object; determine an overall representative depth value of the corresponding group based on bounding box representative depth values of clustered objects; and exclude the clustered objects from identified objects in the image upon a determination that the overall representative depth value is greater than a threshold.

First claim

What is claimed is:

1 . A system for processing captured images of objects at a product storage facility, the system comprising:
a trained machine learning model stored in a memory; and a control circuit executing the trained machine learning model configured to:
obtain a captured image of a plurality of objects at the product storage facility;
cluster objects depicted in the captured image that have same product identifiers into a corresponding group;
determine coordinates of each bounding box of each clustered object in the corresponding group;
determine a bounding box representative depth value of pixels inside the bounding box of each clustered object in the corresponding group;
determine an overall representative depth value of the corresponding group based on bounding box representative depth values of clustered objects in the corresponding group; and
exclude the clustered objects from identified objects in the captured image upon a determination that the overall representative depth value is greater than a threshold.

2 . The system of claim 1 , further comprising:
one or more image capture devices configured to capture images of objects at the product storage facility; and a database configured to store the images.

3 . The system of claim 2 , wherein at least one of the one or more image capture devices is coupled to a motorized robotic unit.

4 . The system of claim 2 , wherein the captured image obtained by the control circuit comprises images that have gone through object detection or object classification by the control circuit.

5 . The system of claim 1 , wherein the plurality of objects comprise items for commercial sale.

6 . The system of claim 1 , wherein the control circuit is further configured to calculate one or both of an average depth value and a median depth value of the pixels inside the bounding box of each clustered object to determine the bounding box representative depth value of pixels.

7 . The system of claim 1 , wherein the excluded clustered object comprises a background object outside of an aisle of interest.

8 . The system of claim 1 , wherein the captured image is captured by an image capture device that is configured to focus on an area located in an aisle of interest.

9 . The system of claim 8 , wherein the area comprises at least one of a shelf, a pallet, a bin, and/or a rack in the product storage facility.

10 . The system of claim 1 , wherein the control circuit is further configured to determine a location of remaining objects in the clustered objects after an exclusion of other given objects in the clustered objects, wherein the given objects are those objects whose representative depth values are greater than the threshold relative to the other representative depth values of the other objects in the group.

11 . A method for processing captured images of objects at a product storage facility, the method comprising:
obtaining, by a control circuit executing a trained machine learning model stored in a memory, a captured image of a plurality of objects at the product storage facility; clustering, by the control circuit executing the trained machine learning model, objects depicted in the captured image that have same product identifiers into a corresponding group; determining, by the control circuit executing the trained machine learning model, coordinates of each bounding box of each clustered object in the corresponding group; determining, by the control circuit executing the trained machine learning model, a bounding box representative depth value of pixels inside the bounding box of each clustered object in the corresponding group; determining, by the control circuit executing the trained machine learning model, an overall representative depth value of the corresponding group based on bounding box representative depth values of clustered objects in the corresponding group; and excluding, by the control circuit executing the trained machine learning model, the clustered objects from identified objects in the captured image upon a determination that the overall representative depth value is greater than a threshold.

12 . The method of claim 11 , further comprising:
capturing, by one or more image capture devices, images of objects at the product storage facility; and storing, by a database, the images.

13 . The method of claim 12 , wherein at least one of the one or more image capture devices is coupled to a motorized robotic unit.

14 . The method of claim 12 , wherein the captured image obtained by the control circuit comprises images that have gone through object detection or object classification by the control circuit.

15 . The method of claim 11 , wherein the plurality of objects comprise items for commercial sale.

16 . The method of claim 11 , further comprising calculating, by the control circuit, one or both an average depth value or a median depth value of the pixels inside the bounding box of each clustered object to determine the bounding box representative depth value of pixels.

17 . The method of claim 11 , wherein the excluded clustered object comprises a background object outside of an aisle of interest.

18 . The method of claim 11 , wherein the captured image is captured by an image capture device that is configured to focus on an area located in an aisle of interest.

19 . The method of claim 18 , wherein the area comprises at least one of a shelf, a pallet, a bin, and/or a rack in the product storage facility.

20 . The method of claim 11 , further comprising determining, by the control circuit, a location of remaining objects in the clustered objects after an exclusion of other given objects in the clustered objects, wherein the given objects are those objects whose representative depth values are greater than the threshold relative to the other representative depth values of the other objects in the group.

Assignees

• Walmart Apollo Llc

Inventors

• Zhang Han
• Chen Yilun
• Zhang Lingfeng
• Cantor Adam
• Jade Avinash M
• Ellison Benjamin R
• Robinson Jr William Craig
• Yuan Mingquan
• Duan Zhaoliang
• Wang Wei

Classifications

• G06T7/70 Primary

  Determining position or orientation of objects or cameras (camera calibration G06T7/80) · CPC title

• G06T7/50

  Depth or shape recovery · CPC title

• G06V20/52

  Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69) · CPC title

• G06V10/762

  using clustering, e.g. of similar faces in social networks · CPC title

• G06V10/25 Primary

  Determination of region of interest [ROI] or a volume of interest [VOI] · CPC title