Systems and methods for detecting support members of product storage structures at product storage facilities

Patent metadata

Field	Value
Publication number	US-2024257380-A1
Application number	US-202318103338-A
Country	US
Kind code	A1
Filing date	Jan 30, 2023
Priority date	Jan 30, 2023
Publication date	Aug 1, 2024
Grant date	—

Abstract

Systems and methods of detecting support members of product storage structures that store products at a product storage facility include an image capture device that captures images of a product storage structure including vertical and horizontal support members. A computing device including a control circuit is configured to: obtain the images of the product storage structure captured by the image capture device, stitch the obtained images together to generate a stitched image that depicts the product storage structure, generate a color distribution map of the stitched image of the product storage structure to detect individual ones of the horizontal and vertical support members of the product storage structure.

First claim

What is claimed is:

1 . A system for detecting support members of product storage structures at a product storage facility, the system comprising:
an image capture device having a field of view that includes at least a part of a product storage structure at the product storage facility having the products arranged thereon,
wherein the image capture device is configured to capture a plurality of images of the product storage structure, each of the images depicting a portion of the product storage structure; and
wherein the product storage structure includes a plurality of horizontal support members interconnected by plurality of vertical support members; and
a computing device including a control circuit, the computing device being communicatively coupled to the image capture device, the control circuit being configured to:
obtain the plurality of images of the product storage structure captured by the image capture device; stitch the obtained images together to generate a stitched image that depicts at least a portion of the product storage structure; and generate a color distribution map of the stitched image of the product storage structure to detect individual ones of the horizontal support members and individual ones of the vertical support members of the product storage structure.

2 . The system of claim 1 , wherein the image capture device comprises a motorized robotic unit that includes wheels that permit the motorized robotic unit to move about the product storage facility, and a camera to permit the motorized robotic unit to capture the images of the product storage structure.

3 . The system of claim 1 , wherein the control circuit is programmed to generate the stitched image by stitching together multiple pairs of the images having overlapping fields of view.

4 . The system of claim 3 , wherein the control circuit is programmed to use boundary-extending Laplacian blending to stitch together the multiple pairs of the images having the overlapping fields of view to generate the stitched image.

5 . The system of claim 1 , wherein the control circuit is programmed to generate the color distribution map by setting a first threshold range to detect the individual ones of the vertical support members and setting a different second threshold range to detect the individual ones of the horizontal support members.

6 . The system of claim 5 ,
wherein the individual ones of the vertical support members have a first color and the individual ones of the horizontal support members have a second color different from the first color; and wherein the control circuit is programmed to set the first threshold range to detect the first color but not the second color and to set the second threshold range to detect the second color but not the first color.

7 . The system of claim 1 , wherein the control circuit is programmed to analyze the color distribution map via a scan array union find (SAUF) algorithm to detect largest connected objects, wherein the largest connected objects represent the individual ones of the vertical support members and the individual ones of the horizontal support members.

8 . The system of claim 7 , wherein, after detection of the largest connected objects representing the individual ones of the vertical support members and the individual ones of the horizontal support members, the control circuit is programmed to determine respective centroid points of the individual ones of the vertical and horizontal support members, and, based on the centroid points, calculate a positional coordinate of each intersection of a vertical support member with a horizontal support member.

9 . The system of claim 8 , wherein the control circuit is programmed to analyze the calculated positional coordinate of each intersection of a vertical support member with a horizontal support member to calculate a total number of intersection points between the vertical and horizontal support members and, based on the calculated total number of the intersection points, to determine relative locations of the vertical and horizontal support members of the product storage structure.

10 . The system of claim 9 , wherein, based on a determination of the relative locations of the vertical and horizontal support members of the product storage structure, the control circuit is programmed to generate a data set in association with the stitched image of the product storage structure, the data set representing the positional coordinates of the individual ones of the vertical support members of the product storage structure and the positional coordinates of the individual ones of the horizontal support members of the product storage structure.

11 . A method of detecting support members of product storage structures at a product storage facility, the method comprising:
capturing one or more images of a product storage structure of the product storage facility via an image capture device having a field of view that includes the product storage structure, the product storage structure having products arranged thereon and including a plurality of horizontal support members interconnected by plurality of vertical support members, and each of the images depicting a portion of the product storage structure; by a computing device including a control circuit and communicatively coupled to the image capture device:
obtaining the plurality of images of the product storage structure captured by the image capture device;
stitching the obtained images together to generate a stitched image that depicts at least a portion of the product storage structure; and
generating a color distribution map of the stitched image of the product storage structure to detect individual ones of the horizontal support members and individual ones of the vertical support members of the product storage structure.

12 . The method of claim 11 , wherein the image capture device comprises a motorized robotic unit that includes wheels that permit the motorized robotic unit to move about the product storage facility, and a camera to permit the motorized robotic unit to capture the images of the product storage structure.

13 . The method of claim 11 , wherein the control circuit is programmed to generate the stitched image by stitching together multiple pairs of the images having overlapping fields of view.

14 . The method of claim 13 , further comprising, by the control circuit, using boundary-extending Laplacian blending to stitch together the multiple pairs of the images having the overlapping fields of view to generate the stitched image.

15 . The method of claim 11 , wherein the control circuit is programmed to generate the color distribution map by setting a first threshold range to detect the individual ones of the vertical support members and setting a different second threshold range to detect the individual ones of the horizontal support members

16 . The method of claim 15 ,
wherein the individual ones of the vertical support members have a first color and the individual ones of the horizontal support members have a second color different from the first color; and further comprising, by the control circuit, setting the first threshold range to detect the first color but not the second color and setting the second threshold range to detect the second color but not the first color.

17 . The method of claim 11 , further comprising, by the control circuit, analyzing the color distribution map via a scan array union fi

Assignees

• Walmart Apollo Llc

Inventors

• Wang Wei
• Zhang Lingfeng
• Zhang Han
• Jade Avinash M
• Yuan Mingquan
• Duan Zhaoliang
• Chakraborty Siddhartha
• Ellison Benjamin R
• Robinson Jr William Craig
• Rader Eric W

Classifications

• G06T7/70 Primary

  Determining position or orientation of objects or cameras (camera calibration G06T7/80) · CPC title

• G06T7/90

  Determination of colour characteristics · CPC title

• G06V20/52

  Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69) · CPC title

• B25J5/007

  mounted on wheels · CPC title

• G06V10/16

  using multiple overlapping images; Image stitching · CPC title