System and method for populating a virtual shopping cart based on video of a customer's shopping session at a physical store

Patent metadata

Field	Value
Publication number	US-12086872-B2
Application number	US-202217652069-A
Country	US
Kind code	B2
Filing date	Feb 22, 2022
Priority date	Oct 25, 2019
Publication date	Sep 10, 2024
Grant date	Sep 10, 2024

Abstract

An apparatus includes a display, interface, and processor. The interface receives video from a camera located in a physical store and directed at a first physical rack. The camera captures video of the rack during a shopping session. The processor displays a first virtual rack that emulates the first physical rack and includes first and second virtual shelves. The virtual shelves include virtual items, which include graphical representations of physical items located on the physical rack. The processor displays the rack video, which depicts an event including the person interacting with the first physical rack. The processor also displays a virtual shopping cart. The processor receives information associated with the event, identifying the first virtual item. The rack video depicts that the person selected the first physical item while interacting with the first physical rack. The processor then stores the first virtual item in the virtual shopping cart.

First claim

What is claimed is:

1. An apparatus comprising:
a hardware processor configured to:
display a first virtual rack configured to emulate a first physical rack;
display a video that depicts an event comprising a person interacting with the first physical rack during a shopping session;
display a virtual shopping cart;
receive information associated with the event, the information identifying a first virtual item corresponding to a first physical item, wherein the video depicts that the person selected the first physical item while interacting with the first physical rack, wherein receiving the information associated with the event comprises:
detecting that a slider on a slider bar associated with the video has reached a marker on the slider bar; and,
in association with the event, determining a position of the marker based at least in part upon sensor data received from a camera directed to the first physical rack, wherein the marker comprises at least one of a first identification number indicating the first physical rack or a second identification number indicating the first virtual rack;

in response to receiving the information associated with the event, store the first virtual item in the virtual shopping cart; and
in response to determining that the shopping session has ended, compare the virtual shopping cart to an algorithmic shopping cart comprising items determined by an algorithm to have been selected during the shopping session, the algorithmic shopping cart determined based on a set of inputs received from sensors located within a physical store, wherein:
the virtual shopping cart is determined independently from the algorithmic shopping cart; and
the algorithmic shopping cart is generated based at least in part upon input from a weight sensor.

2. The apparatus of claim 1 , wherein the hardware processor is further configured to:
display a second virtual rack configured to emulate a second physical rack;
display a second video that depicts a second event comprising the person interacting with the second physical rack;
receive information associated with the second event, the information identifying a second virtual item corresponding to a second physical item, wherein the second video depicts that the person selected the second physical item while interacting with the second physical rack; and
in response to receiving the information associated with the second event, store the second virtual item in the virtual shopping cart.

3. The apparatus of claim 2 , wherein the hardware processor is further configured to:
determine that the virtual shopping cart matches the algorithmic shopping cart; and
in response to determining that the virtual shopping cart matches the algorithmic shopping cart:
generate a receipt comprising a first price assigned to the first virtual item and a second price assigned to the second virtual item; and
send the receipt to the person.

4. The apparatus of claim 1 , wherein the first virtual rack comprises a first virtual shelf comprising the first virtual item that comprises a graphical representation of the first physical item located on a first physical shelf of the first physical rack.

5. The apparatus of claim 4 , wherein the sensors comprise an array of algorithmic cameras and in response to determining that the algorithmic shopping cart comprises an error, the hardware processor is further configured to:
determine a subset of the set of inputs associated with the error, wherein the subset comprises inputs received from the array of algorithmic cameras;
attach metadata to the subset, the metadata explaining the discrepancy; and
use the subset to train the algorithm.

6. The apparatus of claim 2 , wherein the hardware processor is further configured to:
determine that the virtual shopping cart does not match the algorithmic shopping cart; and
in response to determining that the virtual shopping cart does not match the algorithmic shopping cart:
determine that the algorithmic shopping cart comprises an error; and
in response to determining that the algorithmic shopping cart comprises an error:
generate a receipt based on the virtual shopping cart, the receipt comprising a first price assigned to the first virtual item and a second price assigned to the second virtual item; and
send the receipt to the person.

7. The apparatus of claim 1 , wherein the hardware processor is further configured to:
determine that the virtual shopping cart does not match the algorithmic shopping cart; and
in response to determining that the virtual shopping cart does not match the algorithmic shopping cart:
determine that the virtual shopping cart comprises an error; and
in response to determining that the virtual shopping cart comprises an error:
generate a receipt based on the algorithmic shopping cart, the receipt comprising prices of the items determined by the algorithm to have been selected by the person during the shopping session; and
send the receipt to the person.

8. The apparatus of claim 1 , wherein the information associated with the event comprises information associated with dragging and dropping the first virtual item from the first virtual rack into the first virtual shopping cart.

9. A method comprising:
displaying a first virtual rack configured to emulate a first physical rack;
displaying a video that depicts an event comprising a person interacting with the first physical rack during a shopping session;
displaying a virtual shopping cart;
receiving information associated with the event, the information identifying a first virtual item corresponding to a first physical item, wherein the video depicts that the person selected the first physical item while interacting with the first physical rack, wherein receiving the information associated with the event comprises:
detecting that a slider on a slider bar associated with the video has reached a marker on the slider bar; and
in association with the event, determining a position of the marker based at least in part upon sensor data received from a camera directed to the first physical rack, wherein the marker comprises at least one of a first identification number indicating the first physical rack or a second identification number indicating the first virtual rack;

in response to receiving the information associated with the event, storing the first virtual item in the virtual shopping cart, and
in response to determining that the shopping session has ended, comparing the virtual shopping cart to an algorithmic shopping cart comprising items determined by an algorithm to have been selected during the shopping session, the algorithmic shopping cart determined based on a set of inputs received from sensors located within a physical store, wherein:
the virtual shopping cart is determined independently from the algorithmic shopping cart; and
the algorithmic shopping cart is generated based at least in part upon input from a weight sensor.

10. The method of claim 9 , further comprising:
displaying a second virtual rack configured to emulate a second physical rack;
displaying a second video that depicts a second event comprising the person interacting with a second physical rack;
receiving information associated with the second event, the information identifying a second virtual item corresponding to a second physical item, wherein the second video depicts that the person selected the second physical item while interacting with the second physical rack; and
in response to receiving the information associated with the second event, storing the second virtual item in the virtual shopping cart.

11. The method of claim 10 , further comprising:
determ

Assignees

• 7 Eleven Inc

Inventors

• Magee Matthew Raymond
• Mirza Shahmeer Ali
• Berry Joshua E
• NGUYEN Trong Nghia
• Kurva Ravi Kumar
• Vakacharla Sarath
• Mohan Ranganathan
• Suri Maninder Singh
• Hodge Jonathan Christopher

Classifications

• H04N7/181

  for receiving images from a plurality of remote sources · CPC title

• G09G2354/00

  Aspects of interface with display user · CPC title

• G09G5/14

  Display of multiple viewports · CPC title

• G06F3/0486

  Drag-and-drop · CPC title

• G06F3/147

  using display panels · CPC title