Systems and methods to check-in shoppers in a cashier-less store

Patent metadata

Field	Value
Publication number	US-11538186-B2
Application number	US-202117383303-A
Country	US
Kind code	B2
Filing date	Jul 22, 2021
Priority date	Aug 7, 2017
Publication date	Dec 27, 2022
Grant date	Dec 27, 2022

Abstract

Systems and techniques are provided for linking subjects in an area of real space with user accounts. The user accounts are linked with client applications executable on mobile computing devices. A plurality of cameras are disposed above the area. The cameras in the plurality of cameras produce respective sequences of images in corresponding fields of view in the real space. A processing system is coupled to the plurality of cameras. The processing system includes logic to determine locations of subjects represented in the images. The processing system further includes logic to match the identified subjects with user accounts by identifying locations of the mobile computing devices executing client applications in the area of real space and matching locations of the mobile computing devices with locations of the subjects.

First claim

What is claimed is:

1. A method for linking subjects in an area of real space with corresponding user accounts, the user accounts being linked with client applications executable on mobile computing devices, the method including:
receiving a sequence of images of the area of real space;
identifying one or more subjects in the sequence of images, and determining velocities of the one or more subjects in the sequence of images;
determining velocities of one or more mobile computing devices in the area of real space, based on data transmitted by the one or more mobile computing devices;
comparing (i) the determined velocities of the one or more subjects in the sequence of images and (ii) the determined velocities of the one or more mobile computing devices;
based on the comparing, matching a first mobile computing device of the one or more mobile computing devices with a first subject of the one or more subjects; and
linking the first subject with a first user account linked to a first client application being executed on the first mobile device.

2. The method of claim 1 , wherein determining velocities of one or more mobile computing devices comprises:
receiving accelerometer data transmitted by the one or more mobile computing devices; and
based on the accelerometer data, determining velocities of one or more mobile computing devices in the area of real space.

3. The method of claim 2 , wherein determining velocities of one or more mobile computing devices further comprises:
receiving first accelerometer data from the first mobile computing device, the first accelerometer data indicative of acceleration of the first mobile computing device along one or more axes;
using the first accelerometer data from the first mobile computing device over a plurality of time intervals, determining a corresponding plurality of velocities of the first mobile computing device over the corresponding plurality of time intervals; and
determining an average velocity of the first mobile computing device over the plurality of time intervals, based on the plurality of velocities.

4. The method of claim 1 , wherein determining velocities of one or more mobile computing devices comprises:
determining velocities of those mobile computing devices, for which user accounts have not yet been linked with the corresponding subjects.

5. The method of claim 1 , wherein comparing (i) the determined velocities of the one or more subjects in the sequence of images and (ii) the determined velocities of the one or more mobile computing devices comprises:
comparing (i) the determined velocities of the one or more subjects, who have not yet been linked to corresponding user accounts, in the sequence of images and (ii) the determined velocities of the one or more mobile computing devices, associated user accounts of which have not yet been linked to corresponding subjects.

6. The method of claim 1 , wherein determining velocities of the one or more subjects in the sequence of images comprises:
for the first subject of the one or more subjects in the sequence of images, determining changes in positions of joints of the first subject with respect to time using the sequence of images; and
determining a velocity of the first subject, based on the changes in positions of joints of the first subject with respect to time.

7. The method of claim 6 , wherein determining velocities of the one or more subjects in the sequence of images further comprises:
determining a location of a center of mass of the first subject, based on the joints of the first subject; and
determining a velocity of the center of mass of the first subject, based on the changes in positions of joints of the first subject with respect to time.

8. The method of claim 1 , wherein determining velocities of the one or more subjects in the sequence of images further comprises:
for the first subject of the one or more subjects in the sequence of images, determining a velocity of a hand of the first subject holding a mobile computing device.

9. The method of claim 1 , wherein matching the first mobile computing device of the one or more mobile computing devices with the first subject of the one or more subjects comprises:
determining a norm between velocities of the first mobile computing device of the one or more mobile computing devices and the first subject of the one or more subjects; and
in response to the norm being less than a threshold, matching the first mobile computing device of the one or more mobile computing devices with the first subject of the one or more subjects.

10. The method of claim 1 , wherein matching the first mobile computing device of the one or more mobile computing devices with the first subject of the one or more subjects comprises:
determining a first Euclidean distance between velocities of the first mobile computing device of the one or more mobile computing devices and the first subject of the one or more subjects; and
in response to the first Euclidean distance being less than a threshold, matching the first mobile computing device of the one or more mobile computing devices with the first subject of the one or more subjects.

11. The method of claim 1 , wherein matching the first mobile computing device of the one or more mobile computing devices with the first subject of the one or more subjects comprises:
determining Euclidean distances between velocities of various pairs of mobile computing devices and subject, such that a plurality of Euclidean distances are determined corresponding to a plurality of pairs of mobile computing devices and subjects, each pair including (i) a corresponding mobile computing device of the one or more mobile computing devices that have not yet been linked to corresponding one or more subjects, and (ii) a corresponding subject of the one or more subjects that have not yet been linked to corresponding one or more mobile computing devices;
for each pair of the plurality of pairs, incrementing a score counter in response to a corresponding Euclidean distance associated with the pair being below a first threshold, such that a plurality of score counters corresponding to the plurality of pairs is maintained and selectively incremented;
comparing individual score counters with a second threshold, and selecting (i) a highest-score counter with a highest score above the second threshold and (ii) a second-highest score counter with a second highest score,
wherein the highest-score counter is for a first pair that includes the first mobile computing device and the first subject; and

in response to a difference between the highest-score counter for the first pair and the second-highest score counter being higher than a third threshold, matching the first mobile computing device with the first subject.

12. A non-transitory computer readable storage medium impressed with computer program instructions to link subjects in an area of real space with corresponding user accounts, the user accounts being linked with client applications executable on mobile computing devices, the instructions, when executed on a processor, implement a method comprising:
receiving sensor data from the area of real space;
determining velocities of the one or more subjects, based on the sensor data;
determining velocities of one or more mobile computing devices in the area of real space, based on data transmitted by the one or more mobile computing devices;
determining a norm between (i) a velocity of a first mobile computing device of the one or more mobile computing devices and (ii) a velocity of a first subject of the one or more subjects; and
in response to the norm being less than a threshold, li

Assignees

• Standard Cognition Corp

Inventors

• Fisher Jordan E
• Green Warren
• Fischetti Daniel L

Classifications

• H04N23/60

  Control of cameras or camera modules · CPC title

• H04L67/52

  specially adapted for the location of the user terminal · CPC title

• G06T2207/10016

  Video; Image sequence · CPC title

• G06F3/14

  Digital output to display device {; Cooperation and interconnection of the display device with other functional units} · CPC title

• G06Q30/06

  Buying, selling or leasing transactions · CPC title