Counting inventory items using image analysis

Patent metadata

Field	Value
Publication number	US-10169660-B1
Application number	US-201414578021-A
Country	US
Kind code	B1
Filing date	Dec 19, 2014
Priority date	Dec 19, 2014
Publication date	Jan 1, 2019
Grant date	Jan 1, 2019

Abstract

Described is a system for counting stacked items using image analysis. In one implementation, an image of an inventory location with stacked items is obtained and processed to determine the number of items stacked at the inventory location. In some instances, the item closest to the camera that obtains the image may be the only item viewable in the image. Using image analysis, such as depth mapping or Histogram of Oriented Gradients (HOG) algorithms, the distance of the item from the camera and the shelf of the inventory location can be determined. Using this information, and known dimension information for the item, a count of the number of items stacked at an inventory location may be determined.

First claim

What is claimed is:

1. A computing system, comprising:
a processor; and
a memory coupled to the processor and storing program instructions that when executed by the processor cause the processor to at least:
receive from a first camera a first image of an inventory location, wherein the first image includes a representation of a plurality of inventory items located at the inventory location;
determine from an inventory location data store, an item type corresponding to the inventory location;
segment the first image into a plurality of image segments;
select a first histogram of oriented gradients (“HOG”) model for a first image segment of the plurality of image segments and a second HOG model for a second image segment of the plurality of image segments, wherein:
the first HOG model corresponds to the item type and is representative of a second image of an item of the item type obtained at a first distance or at a first item orientation;
the second HOG model corresponds to the item type and is representative of a third image of the item of the item type obtained at a second distance or at a second orientation;

process the first image to:
generate a first plurality of inventory item feature vectors corresponding to the first image segment, and
generate a second plurality of inventory item feature vectors corresponding to the second image segment;

compare the first plurality of inventory item feature vectors with the first HOG model;
compare the second plurality of inventory item feature vectors with the second HOG model; and
count a number of the first plurality of inventory item feature vectors that are substantially similar to the first HOG model and the number of the second plurality of inventory item feature vectors that are substantially similar to the second HOG model, wherein the count is representative of a quantity of items at the inventory location.

2. The computing system of claim 1 , wherein:
the first image segment corresponds to a front portion of the inventory location;
the second image segment corresponds to a second portion of the inventory location;
the first distance corresponds to a first approximate distance between the item when positioned in the front portion of the inventory location and the first camera; and
the second distance corresponds to a second approximate distance between the item when positioned in the second portion of the inventory location and the first camera.

3. The computing system of claim 1 , wherein the program instructions, that when executed by the processor to cause the processor to compare the first plurality of feature vectors with the first HOG model, further include instructions that cause the processor to at least:
compare the first plurality of inventory item feature vectors corresponding to the first image segment with a third HOG model, wherein the third HOG model corresponds to the item type and is representative of a third image of a stack of items of the item type obtained at the first distance.

4. A computing system comprising:
a processor; and
a memory coupled to the processor and storing program instructions that when executed by the processor cause the processor to at least:
receive from a first camera a first image of an inventory location, wherein the first image includes a representation of a plurality of inventory items located at the inventory location;
determine from an inventory location data store, an item type corresponding to the inventory location;
select a first histogram of oriented gradients (“HOG”) model and a second HOG model, wherein:
the first HOG model corresponds to the item type and is representative of a second image of an item of the item type obtained at a first distance or at a first item orientation;
the second HOG model corresponds to the item type and is representative of a third image of the item of the item type obtained at a second distance or at a second orientation;

process the first image to generate a plurality of inventory item feature vectors;
compare the plurality of inventory item feature vectors with each of the first HOG model and the second HOG model;
determine a count of a number of the plurality of inventory item feature vectors that are substantially similar to at least one of the first HOG model or the second HOG model;
alter at least one of a rotation of the first image or a magnification level of the first image;
process the altered first image to generate a second plurality of inventory item feature vectors;
compare each of the second plurality of inventory item feature vectors with each of the first HOG model and the second HOG model; and
determine a second count of a second number of the second plurality of inventory item feature vectors that are substantially similar to at least one of the first HOG model or the second HOG model, wherein a sum of the count and the second count is representative of a quantity of items at the inventory location.

5. The computing system of claim 4 , wherein the rotation of the first image is at least one of approximately forty-five degrees or ninety-degrees.

6. The computing system of claim 4 , wherein the first distance is determined based at least in part on a model feature vector determined to correspond with the feature vector and a defined distance associated with the model feature vector.

7. A computer-implemented method for counting items at an inventory location, comprising:
under control of one or more computing systems configured with executable instructions,
receiving from a camera an image of a plurality of items positioned at an inventory location,
receiving depth information for each pixel of a plurality of pixels included in the image, wherein the depth information for each pixel represents a distance between an object represented in the image and the camera;
generating a point cloud representative of at least a portion of the image and based at least in part on the depth information;
determining a cluster boundary range based at least in part on item information stored in an item information data store, wherein the item information corresponds to the plurality of items positioned at the inventory location, wherein a base of each cluster boundary corresponds to a position of an inventory shelf with respect to each point of the point cloud;
determining a plurality of clusters of points in the point cloud, each cluster representative of at least a portion of an item of the plurality of items; and
determining an item count based at least in part on a number of clusters.

8. The computer-implemented method of claim 7 , wherein the point cloud includes a plurality of points, each point corresponding to a respective pixel of the image and having a depth;
the method further comprising:
determining a position of an item represented by at least a portion of the point cloud based at least in part on a position of a plurality of points with respect to a shelf of the inventory location.

9. The computer-implemented method of claim 7 , wherein:
the item information identifies a length, a width, and a height of an item type of the plurality of items; and
the cluster boundary range is a percentage of at least one of the length, the width, and the height.

10. The computer-implemented method of claim 7 , wherein receiving from the camera includes, receiving image data that includes depth information generated based at least in part on a comparison of a plurality of images of items positioned at an inventory location.

11. The computer-implemented method of claim 7 , further comprising:
mounting the camera to an underneath side of a shelf; and
orienting the camera toward the inventory locatio

Assignees

• Amazon Tech Inc

Inventors

• Ren Xiaofeng
• MISRA Avishkar
• Manyam Ohil Krishnamurthy
• Bo Liefeng
• Raghavan Sudarshan Narasimha
• Towers Christopher Robert
• Gopal Gopi Prashanth
• Asmi Yasser Baseer

Classifications

• G06V20/52

  Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69) · CPC title

• G06Q10/087 Primary

  Inventory or stock management, e.g. order filling, procurement or balancing against orders · CPC title

• G06F18/23

  Clustering techniques · CPC title

• G06V10/758

  Involving statistics of pixels or of feature values, e.g. histogram matching · CPC title

• G06K9/6218

  Physics · mapped topic