Systems and methods for deep learning-based shopper tracking

Patent metadata

Field	Value
Publication number	US-11270260-B2
Application number	US-202117195495-A
Country	US
Kind code	B2
Filing date	Mar 8, 2021
Priority date	Aug 7, 2017
Publication date	Mar 8, 2022
Grant date	Mar 8, 2022

Abstract

Systems and techniques are provided for tracking puts and takes of inventory items by subjects in an area of real space. A plurality of cameras with overlapping fields of view produce respective sequences of images of corresponding fields of view in the real space. In one embodiment, the system includes first image processors, including subject image recognition engines, receiving corresponding sequences of images from the plurality of cameras. The first image processors process images to identify subjects represented in the images in the corresponding sequences of images. The system includes second image processors, including background image recognition engines, receiving corresponding sequences of images from the plurality of cameras. The second image processors mask the identified subjects to generate masked images. Following this, the second image processors process the masked images to identify and classify background changes represented in the images in the corresponding sequences of images.

First claim

What is claimed is:

1. A method for identifying customer selection of an inventory item, the method comprising:
using a first camera to produce first one or more images of a customer holding an inventory item during a first time period in a first field of view of the first camera;
using a second camera to produce second one or more images of the customer holding the inventory item during the first time period in a second field of view of the second camera, the first and the second field of views at least partially overlapping, and the customer holding the inventory item during the first time period is within an overlapping region of the first and the second field of views;
generating, based on the first one or more images and independent of the second one or more images, a first evaluation of the inventory item, wherein the first evaluation of the inventory item comprises at least (i) a first confidence level of the inventory item being a first inventory item and (ii) a second confidence level of the inventory item being a second inventory item;
generating, based on the second one or more images and independent of the first one or more images, a second evaluation of the inventory item, the second evaluation of the inventory item comprises at least (i) a third confidence level of the inventory item being the first inventory item and (ii) a fourth confidence level of the inventory item being the second inventory item;
determining a final evaluation of the inventory item, based on (i) the first evaluation comprising the first confidence level and the second confidence level and (ii) the second evaluation comprising the third confidence level and the fourth confidence level, wherein the final evaluation of the inventory item comprises an identification of the inventory item held by the customer to be one of the first or second inventory items; and
triggering, at least in part in response to the final evaluation of the inventory item, an action.

2. A method for identifying customer selection of an inventory item, the method comprising:
using first one or more sensors to produce first one or more outputs indicative of an inventory item selected by a customer;
using second one or more sensors to produce second one or more outputs indicative of the inventory item selected by the customer;
generating, based on the first one or more outputs and independent of the second one or more outputs, a first evaluation of the inventory item, wherein the first evaluation of the inventory item comprises a first confidence level of the inventory item being a first inventory item;
generating, based on the second one or more outputs and independent of the first one or more outputs, a second evaluation of the inventory item, wherein the second evaluation of the inventory item comprises a second confidence level of the inventory item being a second inventory item; and
determining, based on the first evaluation and the second evaluation, a final evaluation of the inventory item, wherein the final evaluation of the inventory item identifies the inventory item to be one of the first inventory item or the second inventory item.

3. The method of claim 2 , wherein the second evaluation of the inventory item further comprises a third confidence level of the inventory item being the first inventory item.

4. The method of claim 2 , wherein the first evaluation of the inventory item comprises a plurality of inventory items and a corresponding plurality of logits, wherein each logit of the plurality of logits is indicative of a probability of the selected inventory item being a corresponding inventory item of the plurality of inventory items, wherein the first confidence level associated with a first logit is indicative of a first probability of the selected inventory item being the first inventory item of the plurality of inventory items.

5. The method of claim 4 , wherein the second evaluation of the inventory item comprises the plurality of inventory items and another corresponding plurality of logits, wherein each logit of the other plurality of logits is indicative of another probability of the selected inventory item being a corresponding inventory item of the plurality of inventory items.

6. The method of claim 2 , wherein the first one or more sensors comprise one or more image capturing devices, and the first one or more outputs comprise one or more images.

7. The method of claim 6 , wherein the second one or more sensors are one or more other image capturing devices, and the second one or more outputs comprise one or more other images.

8. The method of claim 2 , wherein the production of the first one or more outputs by the first one or more sensors and the production of the second one or more outputs by the second one or more sensors at least in part temporally overlaps.

9. The method of claim 2 , further comprising:
triggering, at least in part in response to the final evaluation of the inventory item, an action, wherein triggering the action comprises one or more of (i) updating a shopping cart list, (ii) notifying a store employee to check an identification of the inventory item selected by the customer, and/or (iii) controlling a checkout process.

10. The method of claim 2 , wherein generating the first evaluation of the inventory item comprises:
utilizing one or more neural network models to generate the first evaluation of the inventory item.

11. A system for identifying customer selection of an inventory item, comprising:
first one or more sensors to generate first one or more sensor outputs indicative of an inventory item selected by a customer;
second one or more sensors to generate second one or more sensor outputs indicative of the inventory item selected by the customer;
logic to (i) process the first one or more sensor outputs to generate a first evaluation of the inventory item, (ii) process the second one or more sensor outputs to generate a second evaluation of the inventory item, and (iii) generate a final evaluation of the inventory item, based on the first evaluation of the inventory item and the second evaluation of the inventory item,
wherein the first evaluation of the inventory item comprises a first confidence level of the inventory item being a first inventory item,
wherein the second evaluation of the inventory item comprises at least one of (i) a second confidence level of the inventory item being the first inventory item and/or (ii) a third confidence level of the inventory item being a second inventory item, and
wherein the final evaluation of the inventory item identifies the inventory item to be either the first or the second inventory item.

12. The system of claim 11 , wherein the first one or more sensors comprise one or more image capturing devices.

13. The system of claim 11 , wherein:
the logic is to generate the first evaluation of the inventory item, independent of generation of the second evaluation of the inventory item; and
the logic is to generate the second evaluation of the inventory item, independent of generation of the first evaluation of the inventory item.

14. The system of claim 11 , wherein:
the first one or more sensors are to generate the first one or more sensor outputs at least in part simultaneously with the second one or more sensors generating the second one or more sensor outputs.

15. The system of claim 13 , wherein the second one or more sensors comprise one or more image capturing devices.

16. The system of claim 13 , wherein the logic comprises one or more neural network models to generate the first evaluation and the second evaluation of the

Assignees

• Standard Cognition Corp

Inventors

• Fisher Jordan E
• Fischetti Daniel L
• Ogle Brandon L
• Novak John F
• Dorman Kyle E
• Kihara Kenneth S
• Lasheras Juan C

Classifications

• G06V10/82

  using neural networks · CPC title

• G06V10/764

  using classification, e.g. of video objects · CPC title

• H04N23/60

  Control of cameras or camera modules · CPC title

• H04N23/90

  Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums · CPC title

• G06F18/24143

  Distances to neighbourhood prototypes, e.g. restricted Coulomb energy networks [RCEN] · CPC title