Systems and methods of identifying individual retail products in a product storage area based on an image of the product storage area

Patent metadata

Field	Value
Publication number	US-12288408-B2
Application number	US-202217963802-A
Country	US
Kind code	B2
Filing date	Oct 11, 2022
Priority date	Oct 11, 2022
Publication date	Apr 29, 2025
Grant date	Apr 29, 2025

Abstract

Systems and methods of monitoring inventory of a product storage facility include an image capture device configured to move about the product storage areas of the product storage facility and capture images of the product storage areas from various angles. A computing device coupled to the image capture device obtains the images of the product storage areas captured by the image capture device and processes the obtained images of the product storage areas to detect individual products captured in the obtained images. Based on detection of the individual products captured in the images, the computing device analyzes each of the obtained images to extract meta data from the packaging the individual products to detect one more keywords and determine the locations of the detected keywords on the packaging, and then utilize this information to predict an identity of the products associated with the packaging.

First claim

What is claimed is:

1. A system for detecting and recognizing products at product storage areas of a product storage facility, the system comprising:
an image capture device having a field of view that includes a product storage area of the product storage facility having products arranged therein, wherein the image capture device is configured to capture one or more images of the product storage area;
a computing device including a control circuit, the computing device being communicatively coupled to the image capture device, the control circuit being configured to:
obtain at least one image of the product storage area captured by the image capture device;
analyze the at least one image of the product storage area to detect individual ones of the products captured in the at least one obtained image;
process the at least one obtained image to:
extract meta data information from each of the individual ones of the products detected in the at least one obtained image;
identify one or more keywords in the extracted meta data information;
determine a location of the one or more keywords on the individual ones of the products; and
based on a determination of the location of the one or more keywords on the individual ones of the products, predict an identity of at least a first product in the at least one obtained image.

2. The system of claim 1 , wherein the image capture device comprises a motorized robotic unit that includes wheels that permit the motorized robotic unit to move about the product storage facility, and a camera to permit the motorized robotic unit to capture the one or more images of the product storage area.

3. The system of claim 1 , wherein the control circuit is programmed to generate virtual boundary lines in the at least one obtained image, wherein each of the virtual boundary lines surrounds one of the products captured in the at least one obtained image.

4. The system of claim 1 , wherein the control circuit is programmed to extract optical character recognition information as part of the meta data information extracted from each of the individual ones of the products detected in the at least one obtained image.

5. The system of claim 4 , wherein the control circuit is programmed to analyze the optical character recognition information to determine:
a height and width of the one or more keywords identified on the individual ones of the products; and
horizontal and vertical coordinates of the one or more keywords with respect to a height of the individual ones of the products.

6. The system of claim 5 , wherein the control circuit is programmed to generate a list of known product identifiers associated with the one or more keywords identified on the individual ones of the products, the known product identifiers being ranked on the list based on frequency of occurrence of the one or more keywords identified on the individual ones of the products.

7. The system of claim 6 , wherein the control circuit is programmed to correlate images of products associated with the known product identifiers on the list to the at least one image captured by the image capture device via a feature vector model configured to identify an exterior color of packaging of each of the individual ones of the products captured in the at least one obtained image.

8. The system of claim 7 , wherein the control circuit is programmed to predict the identity of at least a first product in the at least one obtained image based on a determination that the exterior color of the packaging associated with one of the known product identifiers on the list matches the exterior color of the packaging of one of the individual ones of the products in the at least one image captured by the image capture device.

9. The system of claim 8 , further comprising an electronic database configured to store the one or more images captured by the image capture device, and store the images of products associated with the known product identifiers, the electronic database being communicatively coupled to at least one of the image capture device and the computing device.

10. The system of claim 9 , wherein the control circuit is programmed to, after making a prediction of the identity of the first product to send a signal to the electronic database to cause the first product in the at least one image captured by the image capture device to be associated with a product identifier corresponding to one of the known product identifiers of the packaging that matched the exterior color of the packaging of the first product.

11. A method of detecting and recognizing products at product storage areas of a product storage facility, the method comprising:
capturing, via an image capture device having a field of view that includes a product storage area of the product storage facility having products arranged therein, one or more images of the product storage area;
by a computing device including a control circuit and being communicatively coupled to the image capture device:
obtaining at least one image of the product storage area captured by the image capture device;
analyzing the at least one image of the product storage area to detect individual ones of the products captured in the at least one obtained image;
processing the at least one obtained image to:
extract meta data information from each of the individual ones of the products detected in the at least one obtained image;
identify one or more keywords in the extracted meta data information;
determine a location of the one or more keywords on the individual ones of the products; and

based on a determination of the location of the one or more keywords on the individual ones of the products, predicting an identity of at least a first product in the at least one obtained image.

12. The method of claim 11 , wherein the image capture device comprises a motorized robotic unit that includes wheels that permit the motorized robotic unit to move about the product storage facility, and a camera to permit the motorized robotic unit to capture the one or more images of the product storage area.

13. The method of claim 11 , further comprising, by the control circuit, generating virtual boundary lines in the at least one obtained image, wherein each of the virtual boundary lines surrounds one of the products captured in the at least one obtained image.

14. The method of claim 11 , further comprising, by the control circuit, extracting optical character recognition information as part of the meta data information extracted from each of the individual ones of the products detected in the at least one obtained image.

15. The method of claim 14 , further comprising, by the control circuit, analyzing the optical character recognition information to determine:
a height and width of the one or more keywords identified on the individual ones of the products; and
horizontal and vertical coordinates of the one or more keywords with respect to a height of the individual ones of the products.

16. The method of claim 15 , further comprising, by the control circuit, generating a list of known product identifiers associated with the one or more keywords identified on the individual ones of the products, the known product identifiers being ranked on the list based on frequency of occurrence of the one or more keywords identified on the individual ones of the products.

17. The method of claim 16 , further comprising, by the control circuit, correlating images of products associated with the known product identifiers on the list to the at least one image captured by the image cap

Assignees

• Walmart Apollo Llc

Inventors

• Zhang Lingfeng
• Zhang Han
• Pachauri Abhinav
• Jhunjhunwala Amit
• Ghosh Ashlin
• Jade Avinash Madhusudanrao
• Balusu Raghava
• Muktevi Srinivas
• Yuan Mingquan
• Duan Zhaoliang
• Huang Zhiwei
• Mao Tianyi

Classifications

• G06V20/36

  Indoor scenes · CPC title

• G06Q10/08 Primary

  Logistics, e.g. warehousing, loading or distribution; Inventory or stock management · CPC title

• G06V20/52

  Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69) · CPC title

• G06Q10/087

  Inventory or stock management, e.g. order filling, procurement or balancing against orders · CPC title

• G06V30/153 Primary

  using recognition of characters or words · CPC title