Zero-friction exit experience via frontend computer vision

Patent metadata

Field	Value
Publication number	US-2025217787-A1
Application number	US-202418990189-A
Country	US
Kind code	A1
Filing date	Dec 20, 2024
Priority date	Dec 29, 2023
Publication date	Jul 3, 2025
Grant date	—

Abstract

Examples provide a system and method for identifying unscanned items in real-time during a current transaction using computer vision object detection and recognition models. Images of a checkout area are received from camera(s) situated near the point-of-sale (POS). An object detection and recognition model analyze image data to identify items in a cart and/or on the POS conveyor belt. An item identifier (ID) is predicted for each identified item. As each item is scanned at the POS, the scanned item IDs are mapped to the identified item IDs. When a ready-to-pay signal is received, any identified items which fail to map to a scanned item are identified as unscanned items. A notification is sent to a user interface (UI) device associated with the POS. The notification includes an image of each unscanned item and an instruction to scan the items which were inadvertently unscanned prior to completion of the transaction.

First claim

What is claimed is:

1 . A system for frontend unscanned item identification, the system comprising:
a processor; and a computer-readable medium storing instructions that are operative upon execution by the processor to: obtain an image of a selected cart and a plurality of items associated with the selected cart; identify the plurality of items associated with the selected cart; predict an item identifier (ID) associated with each item in the plurality of items, wherein a set of identified items comprising a plurality of item IDs associated with the plurality of items is generated; obtain item scan data in real time from a point-of-sale (POS) device, the item scan data comprising an item ID associated with each item scanned at the POS device during a current transaction, wherein a set of scanned items comprises an ID for each scanned item associated with the item scan data received from the POS device; map each scanned item ID to an identified item ID in the set of identified items; upon receiving a scan complete signal from the POS device indicating a user is ready to pay for the set of scanned items, identify a set of unscanned items based on mapping of the set of identified items to the set of scanned items, wherein an unscanned item is an item in the set of identified items that fails to map to a corresponding item ID in the set of scanned items; and send a notification to a user interface device associated with the POS device prior to completion of the current transaction, the notification comprising the set of unscanned items and a set of unscanned item images, wherein the user is instructed to scan each item in the set of unscanned items.

2 . The system of claim 1 , wherein the instructions are further operative to:
isolate the selected cart and a conveyor belt in an image; and extract a cropped image of the selected cart and a conveyor belt area within a staffed checkout area.

3 . The system of claim 1 , wherein the instructions are further operative to:
obtain a plurality of images comprising a plurality of carts; track the selected cart through the plurality of images; and place a bounding box around the selected cart in each image in the plurality of images.

4 . The system of claim 1 , wherein the instructions are further operative to:
estimates a depth value of each cart in a plurality of carts within the image; and identify the selected cart corresponding to the current transaction.

5 . The system of claim 1 , wherein the instructions are further operative to:
add a set of indicators to the set of unscanned item images, wherein the set of indicators comprises a bounding box associated with each unscanned item in at least one image.

6 . The system of claim 1 , wherein the instructions are further operative to:
display an image of each unscanned item in the set of unscanned items on a user interface (UI) device one at a time, wherein the user is instructed to scan each item as it is displayed on the UI device, wherein a first image of a first unscanned item is displayed with a first instruction to scan the first unscanned item; and upon receiving scan data for the first unscanned item, display a second image of a second unscanned item with a second instruction to scan the second unscanned item.

7 . The system of claim 1 further comprising:
a set of cameras, the set of cameras comprising a ceiling mounted camera capturing a first set of images associated with a top view of the selected cart and a camera mounted to a portion of the POS device capturing a second set of images associated with a side view of the selected cart, wherein the set of cameras transmits a plurality of images of the selected cart to a computing device via a network.

8 . A method for frontend unscanned item detection, the method comprising:
obtaining an image of a selected cart and a plurality of items associated with the selected cart; identifying the plurality of items associated with the selected cart; predicting an item identifier (ID) associated with each item in the plurality of items, wherein a set of identified items comprising a plurality of item IDs associated with the plurality of items is generated; obtaining item scan data in real time from a point-of-sale (POS) device, the item scan data comprising an item ID associated with each item scanned at the POS device during a current transaction, wherein a set of scanned items comprises an item ID for each scanned item associated with the item scan data received from the POS device; mapping each scanned item ID to an identified item ID in the set of identified items; upon receiving a scan complete signal from the POS device indicating a user is ready to pay for the set of scanned items, identifying a set of unscanned items based on mapping of the set of identified items to the set of scanned items, wherein an unscanned item is an item in the set of identified items that fails to map to a corresponding item ID in the set of scanned items; and sending a notification to a user interface device associated with the POS device prior to completion of the current transaction, the notification comprising the set of unscanned items and a set of unscanned item images, wherein the user is instructed to scan each item in the set of unscanned items.

9 . The method of claim 8 , further comprising:
receiving first scan data associated with a first scanned item from the POS device, the first scan data comprising a first scanned item ID; adding the first scanned item ID to the set of scanned items; receiving second scan data associated with a second scanned item from the POS device, the second scan data comprising a second scanned item ID; and adding the second scanned item ID to the set of scanned items in real-time prior to receiving a payment associated with completion of the current transaction.

10 . The method of claim 8 , further comprising:
obtaining a plurality of images comprising a plurality of carts; tracking the selected cart through the plurality of images; placing a bounding box around the selected cart in each image in the plurality of images; and cropping each image to isolate the selected cart from other objects based on the bounding box around the selected cart in each image.

11 . The method of claim 8 , further comprising:
estimating, by a depth model, a depth value of each cart in a plurality of carts within the image; and identifying the selected cart corresponding to the current transaction based on the depth value, wherein the depth value indicates a proximity of the selected cart to the POS device.

12 . The method of claim 8 , further comprising:
displaying an image of each unscanned item in the set of unscanned items on a UI device one at a time, wherein the user is instructed to scan each item as it is displayed on the UI, wherein a first image of a first unscanned item is displayed with a first instruction to scan the first unscanned item; and upon receiving scan data for a first unscanned item, displaying a second image of a second unscanned item with a second instruction to scan the second unscanned item.

13 . The method of claim 8 , further comprising:
identifying each item on a conveyor belt associated with the POS device based on a top view image of the conveyor belt; predicting an item ID associated with each item on the conveyor belt; and adding the item ID associated with each item on the conveyor belt to the set of identified items.

14 . The method of claim 8 , further comprising:
generating a plurality of images of the selected cart by a set of cameras, the set of cameras comprising a ce

Assignees

• Walmart Apollo Llc

Inventors

• Zhu Feiyun
• Zhang Lingfeng
• Wang Wei
• Jin Yanmei
• Robinson William Craig
• Mitchell Colin Grant
• Zang Huanyu
• Duan Zhaoliang
• Yuan Mingquan
• Schubert Jr Michael Alvin

Classifications

• G06Q20/208 Primary

  Input by product or record sensing, e.g. weighing or scanner processing · CPC title

• G07G1/0063

  with means for detecting the geometric dimensions of the article of which the code is read, such as its size or height, for the verification of the registration · CPC title

• G06V2201/07

  Target detection · CPC title

• G06V20/52

  Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69) · CPC title

• G06V10/25 Primary

  Determination of region of interest [ROI] or a volume of interest [VOI] · CPC title