Counting inventory items using image analysis and depth information

Patent metadata

Field	Value
Publication number	US-9996818-B1
Application number	US-201615135384-A
Country	US
Kind code	B1
Filing date	Apr 21, 2016
Priority date	Dec 19, 2014
Publication date	Jun 12, 2018
Grant date	Jun 12, 2018

Abstract

Described is a system for counting stacked items using image analysis. In one implementation, an image of an inventory location with stacked items is obtained and processed to determine the number of items stacked at the inventory location. In some instances, the item closest to the camera that obtains the image may be the only item viewable in the image. Using image analysis, such as depth mapping or Histogram of Oriented Gradients (HOG) algorithms, the distance of the item from the camera and the shelf of the inventory location can be determined. Using this information, and known dimension information for the item, a count of the number of items stacked at an inventory location may be determined.

First claim

What is claimed is:

1. A computing system, comprising:
a processor; and
a memory coupled to the processor and storing program instructions that when executed by the processor causes the processor to at least:
receive from a first camera a first image of an inventory location, wherein the first image includes a representation of a plurality of inventory items located at the inventory location;
determine from an inventory location data store, an item type corresponding to the inventory location;
select a plurality of histogram of oriented gradients (“HOG”) models corresponding to the item type;
process the first image to generate a plurality of feature vectors, each feature vector representative of at least a portion of an object of an inventory item of the plurality of inventory items represented in the first image;
compare each of the plurality of feature vectors with each of the plurality of HOG models;
determine that a first feature vector representative of a first object of a first inventory item is substantially similar to at least one of the plurality of HOG models;
determine position information representative of a position of the first object represented by the feature vector;
compare the position information with an expected position of the first object, wherein the expected position is on a top of the first inventory item;
determine that the position information of the first object represented by the feature vector corresponds with the expected position of the feature vector; and
increment an inventory count.

2. The computing system of claim 1 , wherein the program instructions, that when executed by the processor to cause the processor to determine an item count, further include instructions that when executed by the processor cause the processor to at least:
not increment the item count if it is determined that the position information does not correspond with the expected position of the first object represented by the feature vector.

3. The computing system of claim 1 , wherein:
the program instruction that when executed by the processor further cause the processor to determine, based on the position information, a height of the first object from a shelf of the inventory location; and
the program instructions that when executed by the processor to cause the processor to compare the position information with the expected position, further include program instructions that cause the processor to at least:
determine a plurality of heights of the first object; and
determine that the position information of the first object represented by the feature vector corresponds with the expected position if it is determined that the plurality of heights of the first object are substantially similar.

4. A computer-implemented method for counting items, comprising:
under control of one or more computing systems configured with executable instructions,
receiving from a camera an image of an inventory location;
determining a first feature from the image, wherein the first feature is potentially representative of an item positioned at the inventory location, and wherein the first feature is represented by a plurality of pixels in the image;
determining a plurality of distances, each distance corresponding to a distance between a pixel of the plurality of pixels and the camera;
determining that the first feature corresponds with a model feature stored in an item information data store for a type of item located at the inventory location;
determining that each of the plurality of distances are substantially similar; and
incrementing an item count for the inventory location based at least in part on the determination that the first feature corresponds with the model feature and the determination that each of the plurality of distances are substantially similar.

5. The computer-implemented method of claim 4 , further comprising:
comparing the first feature with a histogram of oriented gradients (“HOG”) model to determine that the first feature potentially represents an item of an item type associated with the inventory location.

6. The computer-implemented method of claim 5 , further comprising:
generating, using a HOG algorithm, a feature vector representative of the first feature.

7. The computer-implemented method of claim 5 , wherein the HOG model includes a model feature vector representative of a feature of an item of the item type.

8. The computer-implemented method of claim 4 , further comprising:
receiving depth information identifying distances between the camera and each pixel of the image, wherein the depth information is determined based at least in part on a comparison of a plurality of images obtained by the camera.

9. The computer-implemented method of claim 4 , further comprising:
determining a second feature from the image, wherein the second feature is potentially representative of a second item positioned at the inventory location, and wherein the second feature is represented by a second plurality of pixels in the image;
determining a second plurality of distances, each distance of the second plurality of distances corresponding to a distance between a pixel of the second plurality of pixels and the camera; and
determining that the second feature does not correspond with a second model feature stored in an item information data store for the type of item located at the inventory location.

10. The computer-implemented method of claim 4 , wherein the first feature is at least one of a color, a size, a shape, a pattern, a letter, a label, a logo, a texture, a gradient, a reflectivity, an edge of the item, a character, or a symbol.

11. The computer-implemented method of claim 4 , wherein the plurality of distances identify a position of the first feature with respect to the camera.

12. The computer-implemented method of claim 11 , wherein the position of the first feature is substantially similar to an expected position of the first feature with respect to the camera.

13. The computer-implemented method of claim 4 , wherein the image includes a plurality of features, each feature of the plurality of features potentially representative of an item located at the inventory location.

14. The computer-implemented method of claim 4 , wherein the item count identifies a number of items located at the inventory location.

15. A computing system, comprising:
a processor; and
a memory coupled to the processor and storing program instructions that when executed by the processor causes the processor to at least:
receive from a camera an image of an item located at an inventory location,
receive depth information corresponding to a position of the item represented in the image with respect to the camera;
determine a feature vector representative of a feature of the item represented in the image;
compare the feature vector with a model feature vector corresponding to an item type associated with the inventory location;
determine a position of the feature of the item;
compare the position of the feature with an expected position; and
determine that the item corresponds to the item type based at least in part on the comparison of the feature vector with the model feature vector and the comparison of the position of the feature with the expected position.

16. The computing system of claim 15 , wherein:
the feature is visible from at least one surface of the item; and
the expected position corresponds to a position of a surface upon which the feature is expected to be.

17. The computing system of claim 16 , wherein the expec

Assignees

• Amazon Tech Inc

Inventors

• Ren Xiaofeng
• Manyam Ohil Krishnamurthy
• MISRA Avishkar
• Bo Liefeng
• Raghavan Sudarshan Narasimha
• Towers Christopher Robert
• Gopal Gopi Prashanth
• Asmi Yasser Baseer

Classifications

• G06F16/5866

  using information manually generated, e.g. tags, keywords, comments, manually generated location and time information · CPC title

• G06F16/5854

  using shape and object relationship · CPC title

• G06Q10/087 Primary

  Inventory or stock management, e.g. order filling, procurement or balancing against orders · CPC title

• G06V20/52

  Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69) · CPC title

• G06K9/6202

  Physics · mapped topic