Systems and methods for deep learning-based notifications

Patent metadata

Field	Value
Publication number	US-10474993-B2
Application number	US-201916256361-A
Country	US
Kind code	B2
Filing date	Jan 24, 2019
Priority date	Aug 7, 2017
Publication date	Nov 12, 2019
Grant date	Nov 12, 2019

Abstract

Systems and techniques are provided for tracking puts and takes of inventory items by subjects in an area of real space. A plurality of cameras with overlapping fields of view produce respective sequences of images of corresponding fields of view in the real space. In one embodiment, the system includes first image processors, including subject image recognition engines, receiving corresponding sequences of images from the plurality of cameras. The first image processors process images to identify subjects represented in the images in the corresponding sequences of images. The system includes second image processors, including background image recognition engines, receiving corresponding sequences of images from the plurality of cameras. The second image processors mask the identified subjects to generate masked images. Following this, the second image processors process the masked images to identify and classify background changes represented in the images in the corresponding sequences of images.

First claim

What is claimed is:

1. A system for tracking inventory events, in an area of real space, comprising:
a processing system receiving a plurality of sequences of images of respective fields of view in the real space, the processing system including:
logic to process the sequences of images to identify inventory events in the area of real space, and create data structures for the inventory events including a subject identifier identifying the subject performing the inventory event in the area of real space, an item identifier, a classification confidence score for the item, a location represented by positions in three dimensions of the area of real space and a timestamp; and
logic to track identified subjects in the area of real space and to display images representing the identified subjects, and in response to comparison of a classification confidence score for the item identified in the data structure created for a given inventory event with a threshold to change a color of the display image of the subject identified in the data structure.

2. The system of claim 1 , wherein the logic to track identified subjects is performed without use of personal identifying biometric information.

3. The system of claim 1 , wherein the fields of view of images in the sequences of images overlap with the field of view of at least one other image in the sequences of images.

4. The system of claim 1 , including first image recognition engines receiving the sequences of images, which process images to generate first data sets that identify subjects and locations of the identified subjects in the real space.

5. The system of claim 4 , wherein the first image recognition engines comprise convolutional neural networks.

6. The system of claim 4 , including logic to process the first data sets to specify bounding boxes which include images of hands of the identified subjects in images in the sequences of images.

7. The system of claim 6 , including second image recognition engines, receiving the sequences of images, which process the specified bounding boxes in the images to detect the inventory events and to generate the item identifiers for the detected inventory events.

8. The system of claim 1 , including logic to determine a movement of the identified subject in the area of real space towards an exit from the area of real space and generate notifications when the identified subject is within a predetermined distance to the exit.

9. The system of claim 8 , the processing system including logic that generates notifications in response to a payment status for items identified in inventory events for the identified subject.

10. A method for tracking inventory events in an area of real space, the method including:
identifying inventory events in the area of real space using sequences of images of corresponding fields of view in the real space, and creating data structures for the inventory events including a subject identifier identifying the subject performing the inventory event in the area of real space, an item identifier, a classification confidence score for the item, a location represented by positions in three dimensions of the area of real space and a timestamp; and
tracking identified subjects in the area of real space, and changing a color of display image representing the identified subject in a given inventory event in response to comparisons of classification confidence for the item identified in the data structure created for the given inventory event with a threshold.

11. The method of claim 10 , wherein the
tracking identified subjects is performed without the use of personal identifying biometric information.

12. A non-transitory computer readable storage medium impressed with computer program instructions to track inventory events in an area of real space, the instructions when executed on a processor, implement a method comprising:
identifying inventory events in the area of real space using sequences of images of corresponding fields of view in the real space, and creating data structures for the inventory events including a subject identifier identifying the subject performing the inventory event in the area of real space, an item identifier, a classification confidence score for the item, a location represented by positions in three dimensions of the area of real space and a timestamp; and
tracking identified subjects in the area of real space, and changing a color of display image representing the identified subject in a given inventory event in response to comparisons of classification confidence for the item identified in the data structure created for the given inventory event with a threshold.

13. The non-transitory computer readable storage medium of claim 12 , wherein the
tracking identified subjects is performed without the use of personal identifying biometric information.

14. The non-transitory computer readable storage medium of claim 12 , the method further comprising:
receiving the sequences of images of corresponding fields of view in the real space, and processing the images using image recognition engines to identify subjects and locations of the identified subjects in the real space.

Assignees

• Standard Cognition Corp

Inventors

• Fisher Jordan E
• Fischetti Daniel L
• Ogle Brandon L
• Novak John F
• Dorman Kyle E
• Kihara Kenneth S
• Lasheras Juan C

Classifications

• G06V10/82

  using neural networks · CPC title

• G06V10/764

  using classification, e.g. of video objects · CPC title

• G01S3/00

  Direction-finders for determining the direction from which infrasonic, sonic, ultrasonic or electromagnetic waves, or particle emission, not having a directional significance, are being received (position-fixing by co-ordinating a plurality of determinations of direction or position lines G01S5/00) · CPC title

• G06T7/85

  Stereo camera calibration · CPC title

• G01S5/16

  using electromagnetic waves other than radio waves · CPC title