Systems and methods to check-in shoppers in a cashier-less store

Patent metadata

Field	Value
Publication number	US-11200692-B2
Application number	US-202016842382-A
Country	US
Kind code	B2
Filing date	Apr 7, 2020
Priority date	Aug 7, 2017
Publication date	Dec 14, 2021
Grant date	Dec 14, 2021

Abstract

Systems and techniques are provided for linking subjects in an area of real space with user accounts. The user accounts are linked with client applications executable on mobile computing devices. A plurality of cameras are disposed above the area. The cameras in the plurality of cameras produce respective sequences of images in corresponding fields of view in the real space. A processing system is coupled to the plurality of cameras. The processing system includes logic to determine locations of subjects represented in the images. The processing system further includes logic to match the identified subjects with user accounts by identifying locations of the mobile computing devices executing client applications in the area of real space and matching locations of the mobile computing devices with locations of the subjects.

First claim

What is claimed is:

1. A system for linking subjects in an area of real space with user accounts, the user accounts being linked with client applications executable on mobile computing devices, comprising:
a processing system configured to receive a plurality of sequences of images of corresponding fields of view in the real space, the processing system including computer program instructions executable by the processing system to implement:
logic to determine locations of identified subjects represented in the images,
logic to identify locations of mobile computing devices executing client applications in the area of real space, individual client applications executing on individual mobile computing devices linked with corresponding user accounts, and
logic to match the identified subjects represented in the images with user accounts by
(i) matching locations of the mobile computing devices with locations of the subjects, and
(ii) based on matching locations of the mobile computing devices in the area of real space with locations of the identified subjects, matching and linking the identified subjects with user accounts of the mobile devices.

2. The system of claim 1 , wherein the client applications on the mobile computing devices transmit accelerometer data to the processing system, and the logic to match the identified subjects with user accounts uses the accelerometer data transmitted from the mobile computing devices.

3. The system of claim 1 , further including log data structures including a list of inventory items for the identified subjects, the processing system including logic to associate the log data structure for the matched identified subject to the user account for the identified subject.

4. A system for linking subjects in an area of real space with user accounts, the user accounts being linked with client applications executable on mobile computing devices, comprising:
a processing system configured to receive a plurality of sequences of images of corresponding fields of view in the real space, wherein the client applications on the mobile computing devices transmit accelerometer data to the processing system, the processing system including
logic to determine locations of identified subjects represented in the images,
logic to match the identified subjects with user accounts by identifying locations of mobile devices executing client applications in the area of real space, and matching locations of the mobile computing devices with locations of the subjects,
wherein the logic to match the identified subjects with user accounts uses the accelerometer data transmitted from the mobile computing devices, and
wherein the logic to match the identified subjects with user accounts further comprises:
logic to calculate velocity of mobile computing devices using the accelerometer data transmitted from mobile computing devices,
logic to calculate distance between velocities of pairs of mobile computing devices with unmatched client applications and velocities of not yet linked identified subjects wherein the velocities of not yet linked identified subjects are calculated from changes in positions of joints of subjects over time, and
logic to match mobile computing devices with unmatched client applications to not yet linked identified subjects when the distance between the velocity of a mobile computing device and the velocity of a subject is below a first threshold.

5. The system of claim 4 , wherein the logic to match mobile computing devices with unmatched client application to not yet linked identified subjects further including logic to increment a score counter for a pair of a mobile computing device with unmatched client application and a not yet linked identified subject when the distance between the velocity of the mobile computing device and the velocity of the subject is below a first threshold.

6. The system of claim 5 , further comprising:
logic to compare the score counters for all pairs of the mobile computing devices with unmatched client applications and the not yet linked identified subjects with a second threshold and selecting a score counter with a highest score above the second threshold; and
logic to compare the score of the selected score counter with score of a score counter with a second highest score; and matching the mobile computing device with unmatched client application to not yet linked subject corresponding to the pair of mobile computing device and the subject with the selected score counter when a difference between the scores of the selected score counter and the score of the second highest score counter is above a third threshold.

7. The system of claim 6 , further comprising a trained neural network to predict the first threshold, the second threshold and the third threshold.

8. The system of claim 4 , further comprising logic to calculate the velocities of not yet linked identified subjects from changes in center of mass of subjects over time wherein the center of mass of subjects is determined from locations of joints of corresponding subjects.

9. The system of claim 4 , further comprising logic to process the plurality of sequences of images of corresponding fields of view in the real space to determine hand joints of the not yet linked identified subjects holding mobile computing devices and calculating the velocities of the hand joints of subjects over time.

10. A method for linking subjects in an area of real space with user accounts, the user accounts being linked with client applications executable on mobile computing devices, the method including:
receiving a plurality of sequences of images of corresponding fields of view in the real space;
determining locations of identified subjects represented in the images;
identifying locations of mobile computing devices executing client applications in the area of real space, individual client application executing on individual mobile device associated with corresponding user account;
matching locations of the mobile computing devices with locations of the subjects; and
based on matching locations of the mobile devices in the area of real space with locations of the identified subjects, matching and linking the identified subjects with user accounts of the mobile devices.

11. The method of claim 10 , wherein the client applications on the mobile computing devices transmit accelerometer data, and matching locations of the mobile computing devices with locations of the subjects includes using the accelerometer data transmitted from the mobile computing devices.

12. The method of claim 10 , further including associating a log data structure for the matched identified subject to the user account for the identified subject, wherein the log data structure includes a list of inventory items for the identified subject.

13. A method for linking subjects in an area of real space with user accounts, the user accounts being linked with client applications executable on mobile computing devices, the method including:
receiving a plurality of sequences of images of corresponding fields of view in the real space;
determining locations of identified subjects represented in the images; and
matching the identified subjects with user accounts by (i) identifying locations of mobile computing devices executing client applications in the area of real space, and (ii) matching locations of the mobile computing devices with locations of the subjects, wherein the client applications on the mobile computing devices transmit accelerometer data, wherein matching the identified subjects with user accounts further includes:
calculating velocity of mobile devices using

Assignees

• Standard Cognition Corp

Inventors

• Fisher Jordan E
• Green Warren
• Fischetti Daniel L

Classifications

• H04N23/60

  Control of cameras or camera modules · CPC title

• H04N23/90

  Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• H04L65/612

  for unicast · CPC title