Zero-friction exit experience via computer vision

Patent metadata

Field	Value
Publication number	US-2025218199-A1
Application number	US-202418990377-A
Country	US
Kind code	A1
Filing date	Dec 20, 2024
Priority date	Dec 29, 2023
Publication date	Jul 3, 2025
Grant date	—

Abstract

Examples provide a system and method for identifying unpaid items in real-time using computer vision object detection and recognition models. Images of a cart are selected based on proximity of the cart to one or more anchor points. The object detection and recognition models analyze the image data and identify a set of items in the selected cart. An e-receipt including a set of paid items is selected from a plurality of active electronic receipts based on matching the set of identified items to the set of paid items. Any unmatched items are added to a set of predicted unpaid items. When a receipt corresponding to the selected e-receipt is scanned, a notification identifying the set of predicted unpaid items is provided to a user device for display, enabling a user to identify any unpaid items in a basket of items quickly and accurately in real-time at store exit.

First claim

What is claimed is:

1 . A system for zero-friction unpaid item identification via computer vision, the system comprising:
a processor; and
a computer-readable medium storing instructions that are operative upon execution by the processor to:
select an image of a selected cart from a plurality of images of the selected cart using a set of anchor points associated with a field of view of an image capture device; identify a plurality of items associated with the selected cart using the selected image; predict an item identifier (ID) associated with each item in the plurality of items associated with the selected cart, a set of identified items comprising a plurality of item IDs associated with the plurality of items; select a e-receipt associated with the selected cart from a plurality of active e-receipts using a fuzzy matching of a set of paid items included in the selected e-receipt and the set of identified items generated using the selected image in real time, the set of paid items comprising a receipt item ID associated with each item scanned during a transaction associated with the selected e-receipt; map each receipt item ID in the set of paid items to an identified item ID in the set of identified items, wherein an unmapped item in the set of identified items is a predicted unpaid item; upon receiving a verification request signal associated with the selected e-receipt from a scan device indicating a user is ready to exit, generate a notification including a verification result, wherein the verification result includes a set of unpaid items, wherein each predicted unpaid item in the set of unpaid items is associated with a predicted item ID in the set of identified items that fails to map to a corresponding receipt item ID in the set of paid items; and send the notification to a user interface (UI) device associated with the scan device, the notification comprises a first list of items in the set of paid items and a second list of items in the set of unpaid items.

2 . The system of claim 1 , wherein the instructions are further operative to:
select a first image of the selected cart in which the selected cart is located in proximity to a first anchor point within the field of view of the image capture device; and select a second image of the selected cart in which the selected cart is located in proximity to a second anchor point within the field of view of the image capture device, wherein a trained object detection model analyzes the first image and the second image to detect the plurality of items within the selected cart.

3 . The system of claim 1 , wherein the instructions are further operative to:
obtain the plurality of images comprising a plurality of carts; track the selected cart through the plurality of images using object tracking; and place a bounding box around the selected cart in each image in the plurality of images.

4 . The system of claim 1 , wherein the instructions are further operative to:
generate a set of indicators within the selected image of the selected cart associated with the set of unpaid items, wherein each unpaid item is associated with an indicator in the set of indicators.

5 . The system of claim 1 , wherein the set of indicators comprises a bounding box associated with each unpaid item in the set of unpaid items.

6 . The system of claim 1 , wherein the instructions are further operative to:
display a set of images of the set of unpaid items, wherein an image of each unpaid item is included in the set of images displayed on the UI device.

7 . The system of claim 1 further comprising:
a set of cameras, the set of cameras comprising a ceiling mounted camera capturing a set of images associated with a top view of the selected cart, wherein the set of cameras transmits the plurality of images of the selected cart to a computing device via a network.

8 . A method for zero-friction unpaid item detection, the method comprising:
selecting an image of a selected cart from a plurality of images of the selected cart using a set of anchor points associated with a field of view of an image capture device; identifying a plurality of items associated with the selected cart using the selected image; predicting an item identifier (ID) associated with each item in the plurality of items associated with the selected cart, a set of identified items comprising a plurality of item IDs associated with the plurality of items; selecting a e-receipt associated with the selected cart from a plurality of active e-receipts using a fuzzy matching of a set of paid items included in the selected e-receipt and the set of identified items generated using the selected image in real time, the set of paid items comprising a receipt item ID associated with each item scanned during a transaction associated with the selected e-receipt; mapping each receipt item ID in the set of paid items to the predicted item ID in the set of identified items, wherein an unmapped item in the set of identified items is a predicted unpaid item; upon receiving a verification request signal associated with the selected receipt from a scan device indicating a user is ready to exit, generating a notification including a verification result, wherein the verification result includes a set of unpaid items, wherein each predicted unpaid item in the set of unpaid items is associated with the predicted item ID in the set of identified items that fails to map to a corresponding receipt item ID in the set of paid items; and sending the notification to a user interface (UI) device associated with the scan device, the notification comprising the set of paid items and the set of unpaid items.

9 . The method of claim 8 , further comprising:
receiving first scan data associated with a first receipt from the scan device, the first scan data comprising a first receipt ID associated with the first receipt; retrieving a first result including a first set of unpaid items and a first set of paid items associated with a first basket of items; generating a first notification including the first result, wherein the first notification is transmitted to the UI device for presentation to the user in real-time; receiving second scan data associated with a second receipt from the scan device, the second scan data comprising a second receipt ID; retrieving a second result including a second set of unpaid items and a second set of paid items associated with a second basket of items; and generating a second notification including the second result, wherein the second notification is transmitted to the UI device for presentation to the user in real-time.

10 . The method of claim 8 , further comprising:
obtaining images comprising a plurality of carts; tracking the selected cart through the images; placing a bounding box around the selected cart in each image in the images; and cropping each image to isolate the selected cart from other objects based on the bounding box around the selected cart in each image.

11 . The method of claim 8 , further comprising:
generate a list of predicted universal product code (UPC) values associated with each identified item in the set of identified items, wherein each predicted UPC in the list of predicted UPC values is mapped to a UPC associated with an item in the set of paid items obtained from the selected receipt.

12 . The method of claim 8 , further comprising:
displaying an image of each unpaid item in the set of unpaid items on the UI device one at a time, wherein the user is instructed to scan each item as it is displayed on the UI device, wherein an image of a first unpaid item is displayed with a first instructi

Assignees

• Walmart Apollo Llc

Inventors

• Zhu Feiyun
• Wang Wei
• Zhang Lingfeng
• Mitchell Colin Grant
• Jin Yanmei
• Zang Huanyu
• Ma Xin
• Schubert Jr Michael Alvin
• Robinson William Craig
• Duan Zhaoliang
• Yuan Mingquan
• Pandey Nilesh
• Zhang Han
• Shah Ketan
• Lobo Paul Lewis

Classifications

• G06V10/26

  Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion · CPC title

• G06V20/52

  Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69) · CPC title

• G06V20/64 Primary

  Three-dimensional [3D] objects · CPC title