Systems and methods of transforming image data to product storage facility location information

Patent metadata

Field	Value
Publication number	US-12591847-B2
Application number	US-202217963751-A
Country	US
Kind code	B2
Filing date	Oct 11, 2022
Priority date	Oct 11, 2022
Publication date	Mar 31, 2026
Grant date	Mar 31, 2026

Abstract

Some embodiments provide systems comprising: a machine learning model database; a deblur system configured to receive at least a portion of an image comprising a presumed location label captured by the image capture device, and apply at least a deblurring machine learning framework to generate a deblurred label image comprising the presumed location label; a rectification system configured to apply an machine learning transform algorithm to the deblurred label image to generate a rectified label image; an optical character recognition (OCR) system configured to apply a recognition machine learning model to the rectified label image to estimate text; and a location estimation system configured to estimate a location of the presumed location label as a function of the estimated text of the presumed location label relative to known text on known location labels position at respective different known locations within the product storage facility.

First claim

What is claimed is:

1 . An image based retail location confirmation system, comprising:
a machine learning model database storing a set of two or more machine learning models; a deblur system communicatively coupled over a distributed communication network with the machine learning model database, wherein the deblur system is configured to receive a portion of a first image comprising a first presumed location label and a second presumed location label captured by an image capture device configured to capture images within a product storage facility, and apply at least a deblurring machine learning framework to the portion of the first image and generate a first deblurred label image comprising the first presumed location label and the second presumed location label; a rectification system communicatively coupled over the distributed communication network with the machine learning model database, wherein the rectification system is configured to apply a rectification machine learning transform algorithm to justify the first deblurred label image according to a known shape of location labels to thereby generate a first rectified label image; an optical character recognition (OCR) system communicatively coupled over the distributed communication network with the machine learning model database, wherein the OCR system is configured to:
apply a recognition machine learning model to the first rectified label image to estimate text of the first presumed location label and the second presumed location label; and
verify that identified patterns of the first presumed location label and the second presumed location label conform with a known location label pattern for location labels; and
a location estimation system configured to;
select one of the first presumed location label and the second presumed location label as a selected presumed location label based on which one of the first presumed location label and the second presumed location label is assigned a higher prioritized value for determining location; and
estimate a location within the product storage facility of the selected presumed location label as a function of the estimated text of the selected presumed location label relative to known text on known location labels positioned at respective different known locations within the product storage facility.

2 . The system of claim 1 , further comprising:
an image cropping system communicatively coupled over the distributed communication network with the machine learning model database, wherein the image cropping system is configured to apply a trained cropping machine learning model to identify within and extract from the first image the portion of the first image comprising the first presumed location label the second presumed location label.

3 . The system of claim 2 , further comprising:
a blur evaluation system configured to estimate a level of blur of the portion of the first image and enable the deblur system to generate the first deblurred label image when the estimated level of blur has a predefined relationship with a blur threshold.

4 . The system of claim 2 , further comprising:
a confidence evaluation system configured to apply at least one confidence rule to determine a confidence score of an accuracy of the estimated text of the first presumed location label and the second presumed location label; and enable the location estimation system configured to estimate the location of the image capture device when the confidence score has a predefined relationship with a confidence threshold, and prevent the location estimation system from estimating locations of the image capture device when other confidence scores do not have the predefined relationship with the confidence threshold.

5 . The system of claim 4 , wherein the confidence evaluation system in applying the at least one confidence rule is configured to confirm that the estimated text complies with a predefined alphanumeric pattern of multiple alphanumeric characters.

6 . The system of claim 4 , further comprising:
a mobile task system comprising the image capture device, a task system, and a movement system communicatively coupled with the task system, wherein the task system is configured to implement code to control the movement system to control movement of the task system through the product storage facility; wherein the image capture device is configured to capture the images as the task system moves through one or more portions of the product storage facility.

7 . The system of claim 4 , further comprising:
a portable user computing device comprising the image capture device, wherein the portable user computing device is configured to be transported by a user associated with the portable user computing device as the user moves through the product storage facility, and wherein the image capture device is configured to capture the images as the portable user computing device is transported through one or more portions of the product storage facility.

8 . The system of claim 4 , wherein the deblur system applying the deblurring machine learning framework is configured to apply a generative adversarial network (GAN) to the portion of the first image and generate the first deblurred label image comprising the first presumed location label and the second presumed location label.

9 . The system of claim 4 , further comprising:
a blur training database storing numerous training sets of actual images and numerous training sets of artificial images, wherein the numerous training sets of actual images each comprise: an actual image of one of the known location labels and at least one artificially blurred version of the one of the known location labels; and wherein the numerous training sets of artificial images each comprises: an artificially generated image of a representative location label based on known format, font and size of alphanumeric characters included on the known location labels, and at least one artificially blurred version of the artificially generated image; and a machine learning training system communicatively coupled over the distributed communication network with the machine learning model database and the blur training database, wherein the machine learning training system is configured to repeatedly train over time the deblurring machine learning framework utilizing the numerous training sets of actual images and the numerous training sets of artificial images.

10 . A method of confirming locations within a product storage facility, comprising:
storing, in a machine learning model database, a set of two or more machine learning models; receiving images captured by an image capture device configured to capture the images within the product storage facility; receiving a portion of a first image, of the images, wherein the portion of the first image comprises a first presumed location label and a second presumed location label captured by the image capture device; applying a deblurring machine learning framework to the portion of the first image and generating a first deblurred label image comprising the first presumed location label and the second presumed location label; applying a rectification machine learning transform algorithm to justify the first deblurred label image according to a known shape of location labels and thereby generating a first rectified label image; applying a recognition machine learning model to the first rectified label image to:
estimate text of the first presumed location label and the second presumed location label; and
verify that identified patterns of the first presumed location label and the second presumed location label conform with a known location labe

Assignees

• Walmart Apollo Llc

Inventors

• Chen Yilun
• Dalal Ravi Kumar
• Cantor Adam
• Elumalai Monisha
• Ellison Benjamin R

Classifications

• G06T5/73

  Deblurring; Sharpening · CPC title

• G06V30/153

  using recognition of characters or words · CPC title

• G06T2207/20081

  Training; Learning · CPC title

• G06T7/11

  Region-based segmentation · CPC title

• G06T2207/20132

  Image cropping · CPC title