System and method for populating a virtual shopping cart based on video of a customer's shopping session at a physical store

Patent metadata

Field	Value
Publication number	US-2021125269-A1
Application number	US-202017021011-A
Country	US
Kind code	A1
Filing date	Sep 15, 2020
Priority date	Oct 25, 2019
Publication date	Apr 29, 2021
Grant date	—

Abstract

An apparatus includes a display, interface, and processor. The interface receives video from a camera located in a physical store and directed at a first physical rack. The camera captures video of the rack during a shopping session. The processor displays a first virtual rack that emulates the first physical rack and includes first and second virtual shelves. The virtual shelves include virtual items, which include graphical representations of physical items located on the physical rack. The processor displays the rack video, which depicts an event including the person interacting with the first physical rack. The processor also displays a virtual shopping cart. The processor receives information associated with the event, identifying the first virtual item. The rack video depicts that the person selected the first physical item while interacting with the first physical rack. The processor then stores the first virtual item in the virtual shopping cart.

First claim

What is claimed is:

1 . An apparatus comprising:
a hardware processor configured to:
display a first virtual rack configured to emulate a first physical rack;
display a video that depicts an event comprising a person interacting with the first physical rack during a shopping session;
display a virtual shopping cart;
receive information associated with the event, the information identifying a first virtual item corresponding to a first physical item, wherein the video depicts that the person selected the first physical item while interacting with the first physical rack;
in response to receiving the information associated with the event, store the first virtual item in the virtual shopping cart; and
in response to determining that the shopping session has ended, compare the virtual shopping cart to an algorithmic shopping cart comprising items determined by an algorithm to have been selected during the shopping session, the algorithmic shopping cart determined based on a set of inputs received from sensors located within the physical store, wherein the sensors comprise a set of weight sensors, wherein each item of the items determined by the algorithm to have been selected during the shopping session is positioned in a zone of a physical shelf of the physical rack in which a corresponding weight sensor is located.

2 . The apparatus of claim 1 , wherein the processor is further configured to:
display a second virtual rack configured to emulate a second physical rack; display a second video that depicts a second event comprising the person interacting with a second physical rack; receive information associated with the second event, the information identifying a second virtual item corresponding to a second physical item, wherein the second video depicts that the person selected the second physical item while interacting with the second physical rack; and in response to receiving the information associated with the second event, store the second virtual item in the virtual shopping cart.

3 . The apparatus of claim 2 , wherein the processor is further configured to:
determine that the virtual shopping cart matches the algorithmic shopping cart; and in response to determining that the virtual shopping cart matches the algorithmic shopping cart:
generate a receipt comprising a first price assigned to the first virtual item and a second price assigned to the second virtual item; and
send the receipt to the person.

4 . The apparatus of claim 1 , wherein the first virtual rack comprises a first virtual shelf comprising the first virtual item that comprises a graphical representation of the first physical item located on a first physical shelf of the first physical rack.

5 . The apparatus of claim 4 , wherein the sensors comprise an array of algorithmic cameras and in response to determining that the algorithmic shopping cart comprises an error, the processor is further configured to:
determine a subset of the set of inputs associated with the error, wherein the subset comprises inputs received from the array of algorithmic cameras; attach metadata to the subset, the metadata explaining the discrepancy; and use the subset to train the algorithm.

6 . The apparatus of claim 2 , wherein the processor is further configured to:
determine that the virtual shopping cart does not match the algorithmic shopping cart; and in response to determining that the virtual shopping cart does not match the algorithmic shopping cart:
determine that the algorithmic shopping cart comprises an error; and
in response to determining that the algorithmic shopping cart comprises an error:
generate a receipt based on the virtual shopping cart, the receipt comprising a first price assigned to the first virtual item and a second price assigned to the second virtual item; and
send the receipt to the person.

7 . The apparatus of claim 1 , wherein the processor is further configured to:
determine that the virtual shopping cart does not match the algorithmic shopping cart; and in response to determining that the virtual shopping cart does not match the algorithmic shopping cart:
determine that the virtual shopping cart comprises an error; and
in response to determining that the virtual shopping cart comprises an error:
generate a receipt based on the algorithmic shopping cart, the receipt comprising prices of the items determined by the algorithm to have been selected by the person during the shopping session; and
send the receipt to the person.

8 . The apparatus of claim 1 , wherein the information associated with the event comprises information associated with dragging and dropping the first virtual item from the first virtual rack into the first virtual cart.

9 . A method comprising:
displaying a first virtual rack configured to emulate a first physical rack; displaying a video that depicts an event comprising a person interacting with the first physical rack during a shopping session; displaying a virtual shopping cart; receiving information associated with the event, the information identifying a first virtual item corresponding to a first physical item, wherein the video depicts that the person selected the first physical item while interacting with the first physical rack; in response to receiving the information associated with the event, storing the first virtual item in the virtual shopping cart, and in response to determining that the shopping session has ended, comparing the virtual shopping cart to an algorithmic shopping cart comprising items determined by an algorithm to have been selected during the shopping session, the algorithmic shopping cart determined based on a set of inputs received from sensors located within the physical store, wherein the sensors comprise a set of weight sensors, wherein each item of the items determined by the algorithm to have been selected during the shopping session is positioned in a zone of a physical shelf of the physical rack in which a corresponding weight sensor is located.

10 . The method of claim 9 , further comprising:
displaying a second virtual rack configured to emulate a second physical rack; displaying a second video that depicts a second event comprising the person interacting with a second physical rack; receiving information associated with the second event, the information identifying a second virtual item corresponding to a second physical item, wherein the second video depicts that the person selected the second physical item while interacting with the second physical rack; and in response to receiving the information associated with the second event, storing the second virtual item in the virtual shopping cart.

11 . The method of claim 10 , further comprising:
determining that the virtual shopping cart matches the algorithmic shopping cart; and in response to determining that the virtual shopping cart matches the algorithmic shopping cart:
generating a receipt comprising a first price assigned to the first virtual item and a second price assigned to the second virtual item; and
sending the receipt to the person.

12 . The method of claim 9 , wherein the first virtual rack comprises a first virtual shelf comprising the first virtual item that comprises a graphical representation of the first physical item located on a first physical shelf of the first physical rack.

13 . The method of claim 12 , wherein the sensors comprise an array of algorithmic cameras and the method further comprising, in response to determining that the algorithmic shopping cart comprises an error:
determining a subset of the set of inputs associated with the error,

Assignees

• 7 Eleven Inc

Inventors

• Magee Matthew Raymond
• Mirza Shahmeer Ali
• Berry Joshua E
• NGUYEN Trong Nghia
• Kurva Ravi Kumar
• Vakacharla Sarath
• Mohan Ranganathan
• Suri Maninder Singh
• Hodge Jonathan Christopher

Classifications

• G09G5/00

  Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators (image data processing or generation, in general G06T) · CPC title

• G06F3/147

  using display panels · CPC title

• G06F3/011

  Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00) · CPC title

• H04N7/181

  for receiving images from a plurality of remote sources · CPC title

• G09G2354/00

  Aspects of interface with display user · CPC title