Systems and methods of image processing and shrinkage evaluation

Patent metadata

Field	Value
Publication number	US-12067549-B2
Application number	US-202217588625-A
Country	US
Kind code	B2
Filing date	Jan 31, 2022
Priority date	Jan 31, 2022
Publication date	Aug 20, 2024
Grant date	Aug 20, 2024

Abstract

Some embodiments provide image evaluation systems and methods, comprising: a plurality of camera systems distributed about a retail facility and each of the plurality of camera systems; and an image processing system configured to receive multiple images over time, process each image comprising: determine, from pixel data, a gradient amplitude and directional component; determine a histogram curve from the gradient amplitudes as a function of the directional component of the pixel data, and identify a key direction relative to a maximum accumulation of the gradient amplitudes; for the gradient amplitudes, of the pixel data, having a corresponding directional component that is within a direction threshold of the key direction, identify a number of local maxima corresponding to the key direction; and determine a quantity of items of the product corresponding to a quantity of the number of local maxima.

First claim

What is claimed is:

1. An image evaluation system, comprising:
a camera system configured to capture images of items of a product over a period of time, the items of the products being stored in a monitored location within a retail facility where the items of the product are accessible and retrievable by customers for purchase, wherein the images capture a customer entering and leaving the monitored location; and
an image processing system communicatively coupled with the camera system, wherein the image processing system is configured to receive the images from the camera system, to determine, from pixel data of the images, gradient amplitudes and directional component(s) associated with the items of the product, to determine a first histogram curve from the gradient amplitudes as a function of the directional component(s), to identify a key direction relative to a maximum accumulation of the gradient amplitudes, to identify local maxima corresponding to the key direction, to determine a change in quantity of the items of the product between the customer entering and leaving the monitored location based on a quantity of the local maxima within a threshold distance from another one of the local maxima, to track movement of the customer throughout the retail facility in response to the change in quantity of the items of the product exceeding a threshold, and to notify an employee to interface with the customer following checkout.

2. The system of claim 1 , further comprising:
point of sale (POS) systems within the retail facility configured to complete sales transactions for customers of items of different products and to communicate sales data of the completed sales transactions; and
a product tracking system communicatively coupled with the image processing system and configured to:
maintain an inventory database comprising inventory quantities as a function of the sales data and product shipping information;
evaluate the inventory quantities relative to the quantity of items of the product received from the image processing system;
identify when there is a threshold discrepancy as a function of the inventory quantities relative to the quantity of items of the product; and
generate a notification of the discrepancy.

3. The system of claim 1 , wherein the image processing system is further configured to:
identify, from the first histogram curve, a key direction relative to a secondary maximum accumulation of the gradient amplitudes;
for the gradient amplitudes, of the pixel data, having a corresponding directional component that is within a second direction threshold of the key direction relative to a secondary maximum accumulation, determine a secondary curve of gradients;
determine, from the secondary curve of gradients, multiple local secondary maxima;
identify, within the image and as a function of the multiple local secondary maxima, packaging boundaries defining multiple sub-sections of a packaging in which the items of the product are positioned;
divide the image into sub-images each corresponding to one of the sub-sections of the packaging identified based on the identified packaging boundaries;
identify, within each sub-image, a subset of a number of local maxima corresponding to the key direction in the respective sub-image;
determine a sub-quantity of the items of the product corresponding to a sub-quantity of the subset of the number of the local maxima of the respective sub-image; and
sum the determined sub-quantities of the items of the product in determining the quantity of items.

4. The system of claim 3 , wherein the multiple sub-sections of the packaging each corresponds to one of multiple columns of the packaging and within each of the columns is contained the respective sub-quantity of the items of the product.

5. The system of claim 3 , wherein the image processing system, in identifying the number of local maxima corresponding to the key direction, is configured to obtain a primary curve of gradients of pixels based on the gradients that have the corresponding directional component that is within a direction threshold of the key direction; and
determine, from the primary curve of gradients, the number of local maxima corresponding to the key direction.

6. The system of claim 5 , wherein the key direction is substantially orthogonal to the key direction relative to a secondary maximum accumulation.

7. The system of claim 1 , wherein the image processing system is further configured to evaluate color data of the pixel data relative to a known color pattern of packaging of the product based on known dimensions of the packaging of the product; and
segment, from within the image based on the evaluation of the color data, the image into multiple different box sub-section images of the image.

8. The system of claim 7 , wherein the image processing system, in determining the quantity of the items of the product, is configured to:
identify, for each box sub-section image, the key direction relative to the maximum accumulation of the gradient amplitudes;
identify, for each box sub-section image of the multiple box sub-section images, a number of local maxima corresponding to the key direction; and
determine multiple boxset quantities of the items of the product each corresponding to one of the box sub-section image of the image, wherein the respective boxset quantity of the items of the product corresponding to respective quantities of a of the local maxima that are each within the threshold distance from another one of a number of local maxima, and sum the multiple boxset quantities to obtain the quantity of items of the product.

9. The system of claim 7 , wherein the image processing system, in evaluating the color data of the pixel data, is configured to:
apply a color clustering to the pixel data of the image based on the known color pattern of the packaging of the product; and
segment the image into the multiple box sub-section images based on the color clustering.

10. A method of processing images, comprising:
receiving, by an image processing system from a camera system, images of items of a product captured by the camera system over a period of time, the items of the products being stored in a monitored location within a retail facility where the items of the product are accessible and retrievable by customers for purchase, wherein the images capture a customer entering and leaving the monitored location;
determining, from pixel data of the images, gradient amplitudes and directional component(s) associated with the items of the product;
determining a first histogram curve from the gradient amplitudes as a function of the directional component(s);
identifying a key direction relative to a maximum accumulation of the gradient amplitudes;
identifying local maxima corresponding to the key direction;
determining a change in quantity of the items of the product between the customer entering and leaving the monitored location based on a quantity of the local maxima within a threshold distance from another one of the local maxima, to track movement of the customer throughout the retail facility in response to the change in quantity of the items of the product exceeding a threshold; and
notifying an employee to interface with the customer following checkout.

11. The method of claim 10 , further comprising:
completing, by point of sale (POS) systems within the retail facility configured to complete sales transactions for customers of items of different products and to communicate sales data of the completed sales transactions;
maintaining, by a product tracking system, an inventory database comprising inventory quantities as a function of the sales

Assignees

• Walmart Apollo Llc

Inventors

• Xiao Zhichun
• Wang Chunmei

Classifications

• G06V10/44

  Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components · CPC title

• G06V10/762

  using clustering, e.g. of similar faces in social networks · CPC title

• G06V10/56

  relating to colour · CPC title

• G06V10/267

  by performing operations on regions, e.g. growing, shrinking or watersheds · CPC title

• G06Q10/087 Primary

  Inventory or stock management, e.g. order filling, procurement or balancing against orders · CPC title