Constructing shopper carts using video surveillance

Patent metadata

Field	Value
Publication number	US-10943128-B2
Application number	US-201816177744-A
Country	US
Kind code	B2
Filing date	Nov 1, 2018
Priority date	Nov 1, 2018
Publication date	Mar 9, 2021
Grant date	Mar 9, 2021

Abstract

A system can visually track which items in a store are selected for purchase by a shopper. The system can form a virtual shopping cart by analyzing multiple images, over time, to determine which purchasable items are located with the shopper, such as in a physical shopping cart, in a basket, or held by the shopper. By analyzing multiple images, over time, the system can account for items misidentified in one or more images, or fully or partially obscured in one or more images as the shopper traverses the store. Alternatively, the system can form a virtual shopping cart by analyzing instances in which a purchasable item is removed from a shelf or placed on a shelf. Items removed from, but not returned to, a shelf can be considered to be selected for purchase. The system can include a frictionless checkout that charges the shopper for the selected items.

First claim

What is claimed is:

1. A method, comprising:
capturing, with video surveillance of a store, an image of a shopper;
determining, using a processor coupled to a video interface, that the shopper is not a person already present in the store;
associating, using the processor, a virtual shopping cart with the shopper;
automatically tracking, the video surveillance of a store, movement of the shopper throughout the store and items in the store selected for purchase by the shopper, wherein the tracking includes determining a historical position of the selected items to determine a velocity of the selected items, wherein, when the velocity of the selected items is in a direction away from a shelf, the tracking includes storing a data representation of the selected items in the virtual shopping cart, and wherein when the velocity of the selected items is in a direction toward the shelf, the tracking includes removing the selected items from the virtual shopping cart; and
charging, through a frictionless check-out of the store, an account associated with the shopper for purchase of the selected items, wherein the frictionless check-out includes reconciling the data representation of the selected items in the virtual shopping cart.

2. The method of claim 1 , wherein automatically tracking the items in the store selected for purchase by the shopper comprises:
capturing images from at least one video stream of a shopping area of the store;
automatically recognizing the shopper in a plurality of the captured images;
visually identifying, from each of the plurality of captured images, purchasable items located within a bounding area around the shopper; and
using a sliding time window for the plurality of captured images to correct purchasable items that are misidentified or obscured in one or more of the plurality of captured images.

3. The method of claim 2 , wherein using the sliding time window comprises:
automatically analyzing a plurality of historical images, the historical images being a specified number of images from the video stream captured immediately prior to a current captured image; and
visually identifying an inventory of any purchasable items located within a bounding area around the shopper, for each historical image.

4. The method of claim 3 , wherein using the sliding time window further comprises:
identifying a misidentified inventory of a first historical image of the plurality of historical images, wherein an inventory of a historical image taken before the first historical image matches an inventory of a historical image taken after the first historical image but differs from the misidentified inventory of the first historical image.

5. The method of claim 4 , wherein using the sliding time window further comprises:
ignoring the misidentified inventory, such that the items selected for purchase match the inventories of the historical images taken before and after the first historical image.

6. The method of claim 5 , wherein the video surveillance of the store comprises:
with at least one overhead camera directed generally downward, capturing at least one video stream of a shopping area of the store.

7. The method of claim 6 , wherein each overhead camera has a field of view that is centered within twenty degrees of a vertical axis.

8. The method of claim 7 , wherein each overhead camera has a field of view that is centered within ten degrees of the vertical axis.

9. A system, comprising:
at least one video camera positioned to capture at least one video stream of a store;
a video interface configured to receive the at least one video stream; and
a processor coupled to the video interface;
memory, including instructions stored thereon, which, when executed by the processor, cause the processor to perform data processing activities, the data processing activities comprising:
capture an image of a shopper;
determine that the shopper is not a person already present in the store;
associating a virtual shopping cart with the shopper;
automatically tracking, from the at least one video stream, movement of the shopper throughout the store and items in the store selected for purchase by the shopper, wherein the tracking includes determining a historical position of the selected items to determine a velocity of the selected items, wherein, when the velocity of the selected items is in a direction awa from a shelf, the tracking includes storing a data representation of the selected items in the virtual shopping cart, and wherein when the velocity of the selected items is in a direction toward the shelf, the tracking includes removing the selected items from the virtual shopping cart; and
charging, through a frictionless check-out of the store, an account associated with the shopper for purchase of the selected items, wherein the frictionless check-out includes reconciling the data representation of the selected items in the virtual shopping cart.

10. The system of claim 9 , wherein:
each video camera has a field of view centered around a downward-facing generally vertical axis; and
automatically tracking the items in the store comprises:
visually identifying an inventory of any purchasable items located within a bounding area around the shopper, for multiple images from the video stream; and
relying on a sliding window of images from the video stream to correct a misidentified inventory of an image from the video stream.

Assignees

• Ncr Corp

Inventors

• Zucker Brent Vance
• Bjelcevic Stefan
• Lieberman Adam Justin

Classifications

• G06V20/52

  Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69) · CPC title

• G06Q30/0635 Primary

  replenishment orders; recurring orders · CPC title

• G06F18/22

  Matching criteria, e.g. proximity measures · CPC title

• H04N23/90

  Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums · CPC title

• G06V40/10

  Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands · CPC title