Realtime inventory location management using deep learning

Patent metadata

Field	Value
Publication number	US-11023850-B2
Application number	US-201916256936-A
Country	US
Kind code	B2
Filing date	Jan 24, 2019
Priority date	Aug 7, 2017
Publication date	Jun 1, 2021
Grant date	Jun 1, 2021

Abstract

Systems and techniques are provided for tracking locations of inventory items in an area of real space including inventory display structures. A plurality of cameras are disposed above the inventory display structures. The cameras in the plurality of cameras produce respective sequences of images in corresponding fields of view in the real space. A memory stores a map of the area of real space identifying inventory locations on inventory display structures. The system is coupled to a plurality of cameras and uses the sequences of images produced by at least two cameras in the plurality of cameras to find a location of an inventory event in three dimensions in the area of real space. The system matches the location of the inventory event with an inventory location.

First claim

What is claimed is:

1. A system for tracking inventory events, such as puts and takes, in an area of real space including inventory display structures, comprising:
memory storing a map of the area of real space, the map identifying inventory locations on inventory display structures in the area of real space; and
a processing system configured to receive a plurality of sequences of images from a corresponding plurality of sensors having corresponding fields of view in the real space including inventory display structures, the field of view of each sequence of images in the plurality of sequences of images overlapping with the field of view of at least one other sequence of images in the plurality of sequences of images, the processing system including image processing logic that uses sequences of images in the plurality of sequences of images to find a location of an inventory event in three dimensions in the area of real space, and logic to match the location of the inventory event with an inventory location;
the image processing logic including for each sequence of images in the plurality of sequences of images, a first neural network processing images in a corresponding sequence of images to produce data specifying locations of subjects in the images, and a second neural network processing images in the corresponding sequence of images to identify items in the images in the specified locations, and
the image processing logic including logic which receives outputs from the first and second neural networks for the plurality of sequences of images to perform time sequence analysis of the outputs to identify the inventory events and their locations.

2. The system of claim 1 , wherein the inventory event is one of a put and take of an inventory item, and the processing system includes logic that updates a log data structure of inventory items associated with the inventory events at the matching inventory location.

3. The system of claim 2 , wherein the log data structure for a given inventory location includes item identifiers and their respective quantities for items identified on the given inventory location.

4. The system of claim 1 , wherein the image processing logic creates a data structure including an item identifier, a put or take indicator, coordinates along three axes of the area of real space and a timestamp.

5. The system of claim 1 , wherein the image processing logic uses the outputs for the first neural network and the sequences of images to generate data sets representing elements in the images corresponding to hands, and the second neural network processes portions of the images corresponding to hands from the corresponding sequence.

6. The system of claim 1 , wherein the first neural network comprises a convolutional neural network.

7. The system of claim 1 , wherein the plurality of sensors are synchronized sensors.

8. The system of claim 1 , further including logic to update a log data structure for the area of real space including items identifiers and their respective quantities in the area of real space.

9. The system of claim 1 , wherein the logic to match the location of the inventory event with an inventory location in the three dimensional map executes a procedure including calculating a distance from the location of the inventory event to inventory locations on inventory display structures and matching the inventory event with an inventory location based on the calculated distance.

10. The system of claim 1 , further including a planogram identifying positions of inventory display structures in the area of real space and items positioned on shelves on the inventory display structures, the processing system including logic to determine misplaced items if the inventory event is matched with an inventory location that does not match the planogram.

11. The system of claim 1 , wherein the first neural generates first classification data classifying elements of the images in a first sequence in the plurality of sequences of images as representing types of joints, and the first neural network that processes a second sequence of images in the plurality of sequences of images generates first classification data classifying elements of the images in the second sequence as representing types of joints; and
the second neural network that processes the first sequence of images generates second classification data classifying elements of the images in the first sequence as representing types of inventory items in hands, and the second neural network that processes the second sequence of images generates second classification data classifying elements of the images in the second sequence as representing types of inventory items in hands.

12. The system of claim 11 , wherein the third logic comprises third neural networks that process outputs of the first logic and the second logic.

13. A method of tracking inventory events, such as puts and takes, in an area of real space including inventory display structures, the method including:
using a plurality of sequences of images from a corresponding plurality of sensors having corresponding fields of view in the real space, including the inventory display structures, the field of view of each sequence of images in the plurality of sequences of images overlapping with the field of view of at least one other sequence of images in the plurality of sequences of images;
finding a location of an inventory event in three dimensions in the area of real space using at least first and second sequences of images having overlapping fields of view in the plurality of sequences of images, including using first neural networks to process the first and second sequences of images to produce data specifying locations of subjects, second neural networks to process the first and second sequences of images to identify items in the specified locations, and time sequence analysis of outputs of the first and second neural networks for the first and second sequences of images to identify the inventory event and the location in three dimensions; and
matching the location of the inventory event with an inventory location.

14. The method of claim 13 , wherein the inventory event is one of a put and a take of an inventory item, and the method further including, updating a log data structure of inventory items associated with the inventory events at the matching inventory location.

15. The method of claim 14 , wherein the log data structure for a given inventory location includes item identifiers and their respective quantities for items identified on the given inventory location.

16. The method of claim 13 , wherein the finding a location of an inventory event in three dimensions in the area of real space includes creating a data structure including an item identifier, a put or take indicator, three dimensional coordinates of the inventory event in the area of real space and a timestamp.

17. The method of claim 13 , including using outputs of the first neural network and the plurality of sequences of images to generate data sets representing elements in the images corresponding to hands, and the second neural networks process the images of hands from the first and second sequences of images.

18. The method of claim 17 , wherein said first neural networks include convolutional neural networks and including combining results from the convolutional neural networks to find the location of the inventory event.

19. The method of claim 13 , wherein sequences of images in the plurality of sequences of images are received from

Assignees

• Standard Cognition Corp

Inventors

• Fisher Jordan E
• Fischetti Daniel L
• Locascio Nicholas J

Classifications

• H04N23/60

  Control of cameras or camera modules · CPC title

• H04N23/90

  Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums · CPC title

• G06N7/01

  Probabilistic graphical models, e.g. probabilistic networks · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title