Systems and methods of selecting an image from a group of images of a retail product storage area

Patent metadata

Field	Value
Publication number	US-12450558-B2
Application number	US-202217963787-A
Country	US
Kind code	B2
Filing date	Oct 11, 2022
Priority date	Oct 11, 2022
Publication date	Oct 21, 2025
Grant date	Oct 21, 2025

Abstract

Systems and methods of monitoring inventory of a product storage facility include an image capture device configured to move about the product storage areas of the product storage facility and capture images of the product storage areas from various angles. A computing device coupled to the image capture device obtains the images of the product storage areas captured by the image capture device and processes the obtained images of the product storage areas to detect individual products captured in the obtained images. Based on detection of the individual products captured in the images, the computing device analyzes each of the obtained images to detect one or more adjacent product storage structures (shelves, pallets, etc.) and identifies and selects a single image that fully shows a product storage structure of interest and fully shows each of the products stored on the product storage structure of interest.

First claim

What is claimed is:

1. A system for monitoring inventory of a product storage facility, the system comprising:
an image capture device having a field of view that includes a product storage area of the product storage facility having products arranged therein, wherein the image capture device is configured to:
move about the product storage area; and
capture a plurality of images of the product storage area from a plurality of viewing angles; and

a computing device including a control circuit, the computing device being communicatively coupled to the image capture device, the control circuit being configured to:
obtain the plurality of images of the product storage area captured by the image capture device;
process each of the obtained images of the product storage area to detect individual ones of each of the products captured in each of the obtained images;
based on detection of the individual ones of each of the products captured in each of the obtained images, identify at least a first product storage structure located in the product storage area that stores a first group of identical products thereon;
calculate, for each pair of consecutively captured images of the plurality of images, location information representing a difference in depicted locations between the pair of consecutively captured images;
generate, based on the location information, a set of images of the plurality of images that each at least partially depict the first product storage structure;
generate a modified set of images comprising the set of images and a plurality of virtual bounding boxes, each virtual bounding box of the plurality of virtual bounding boxes surrounding an individual product depicted in the set of images;
process the modified set of images using a clustering algorithm to determine a first group of bounding boxes of the plurality of virtual bounding boxes representative of the first group of identical products;
based on the first group of bounding boxes, identify and select a single image that fully shows the first product storage structure and fully shows each of the products in the first group of identical products stored on the first product storage structure; and
transmit the single image to an electronic database for use in monitoring inventory at the product storage facility.

2. The system of claim 1 , wherein the image capture device comprises a motorized robotic unit that includes wheels that permit the motorized robotic unit to move about the product storage facility, and a camera to permit the motorized robotic unit to capture the plurality of images of the product storage area from the plurality of viewing angles.

3. The system of claim 2 , wherein the image capture device is configured to transmit, to the computing device, one or more signals including the plurality of images of the product storage area captured by the camera.

4. The system of claim 1 , wherein the control circuit is programmed to cluster the plurality of virtual bounding boxes in each of the obtained images to determine a number of adjacent product storage structures in the product storage area present in each of the obtained images.

5. The system of claim 4 , wherein the adjacent product storage structures are pallets or shelves.

6. The system of claim 4 , wherein the control circuit is programmed to analyze the clustered plurality of virtual bounding boxes in each of the obtained images to identify the first product storage structure located in the product storage area that stores the first group of identical products thereon.

7. The system of claim 6 , wherein the control circuit is programmed to analyze the clustered plurality of virtual bounding boxes in each of the obtained images to identify at least a second product storage structure located in the product storage area that stores a second group of identical products thereon, wherein the products of the second group of identical products are different from the products of the first group of identical products.

8. The system of claim 1 , wherein the control circuit is programmed to not send to the electronic database and discard the obtained images that do not fully show the first product storage structure or do not fully show each of the products in the first group of identical products stored on the first product storage structure.

9. A method of monitoring inventory of a product storage facility, the method comprising:
capturing, from a plurality of viewing angles, a plurality of images of a product storage area of the product storage facility having products arranged therein via an image capture device moving about the product storage area and having a field of view that includes the product storage area; and
by a computing device including a control circuit, the computing device being communicatively coupled to the image capture device:
obtaining the plurality of images of the product storage area captured by the image capture device;
processing each of the obtained images of the product storage area to detect individual ones of each of the products captured in each of the obtained images;
based on detection of the individual ones of each of the products captured in each of the obtained images, identifying at least a first product storage structure located in the product storage area that stores a first group of identical products thereon;
calculating, for each pair of consecutively captured images of the plurality of images, location information representing a difference in depicted locations between the pair of consecutively captured images;
generating, based on the location information, a set of images of the plurality of images that each at least partially depict the first product storage structure;
generating a modified set of images comprising the set of images and a plurality of virtual bounding boxes, each virtual bounding box of the plurality of virtual bounding boxes surrounding an individual product depicted in the set of images;
processing the modified set of images using a clustering algorithm to determine a first group of bounding boxes of the plurality of virtual bounding boxes representative of the first group of identical products;
based on the first group of bounding boxes, identifying and selecting a single image that fully shows the first product storage structure and fully shows each of the products in the first group of identical products stored on the first product storage structure; and
transmitting the single image to an electronic database for use in monitoring inventory at the product storage facility.

10. The method of claim 9 , wherein the image capture device comprises a motorized robotic unit that includes wheels that permit the motorized robotic unit to move about the product storage facility, and a camera to permit the motorized robotic unit to capture the plurality of images of the product storage area from the plurality of viewing angles.

11. The method of claim 10 , further comprising transmitting, from the image capture device to the computing device, one or more signals including the plurality of images of the product storage area captured by the camera.

12. The method of claim 9 , further comprising, by the control circuit, clustering the plurality of virtual bounding boxes in each of the obtained images to determine a number of adjacent product storage structures in the product storage area present in each of the obtained images.

13. The method of claim 12 , wherein the adjacent product storage structures are pallets or shelves.

14. The method of claim 12 , further comprising, by the control circuit, analyzing the clustered plurality of vir

Assignees

• Walmart Apollo Llc

Inventors

• Zhang Lingfeng
• Yuan Mingquan
• Lobo Paul Lewis
• Jade Avinash M
• Xiao Zhichun
• Robinson Jr William Craig
• Duan Zhaoliang
• Wang Wei
• Zhang Han
• Balusu Raghava
• Mao Tianyi

Classifications

• G06V10/25

  Determination of region of interest [ROI] or a volume of interest [VOI] · CPC title

• G06V10/762

  using clustering, e.g. of similar faces in social networks · CPC title

• G06V20/36

  Indoor scenes · CPC title

• G06V20/52

  Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69) · CPC title

• G06V10/16

  using multiple overlapping images; Image stitching · CPC title