Inferring count of items using image

Patent metadata

Field	Value
Publication number	US-10339656-B1
Application number	US-201615280595-A
Country	US
Kind code	B1
Filing date	Sep 29, 2016
Priority date	Sep 29, 2016
Publication date	Jul 2, 2019
Grant date	Jul 2, 2019

Abstract

Images of a fixture such as a shelf holding items may be acquired by cameras and processed to count the quantity of items at the fixture. A top of an item is determined in the image. Given information about the items designated for stowage at the fixture and the location of the top, a three-dimensional (3D) bounding box indicative of a volume is determined relative to the fixture. Bounding boxes which extend outside the boundaries of the fixture are disregarded. Remaining bounding boxes may then be analyzed to determine a measured height of the item(s) in a stack. The measured height may be divided by a per-item height to determine a quantity of items in the stack. The quantities in multiple stacks may be summed to determine a quantity at the fixture.

First claim

What is claimed is:

1. A system comprising:
a memory, storing computer-executable instructions; and
a hardware processor to execute the computer-executable instructions to:
access image data representative of a fixture;
access data indicative of a type of item associated with the fixture;
access two-dimensional (2D) data indicative of at least a portion of a top of the type of item;
access three-dimensional (3D) data associated with the type of item;
determine, using the image data, one or more estimated tops of the type of item at the fixture;
determine one or more estimated item locations associated with the one or more estimated tops of the type of item at the fixture, wherein the one or more estimated item locations are indicative of points in 3D space;
determine a first set of the one or more estimated item locations that comprises those estimated item locations that are within a working volume of the fixture, the working volume comprising a volume where the type of item could be physically present;
determine one or more estimated dimensions based at least in part on the first set of one or more the estimated item locations; and
determine a quantity of the type of item at the fixture using the one or more estimated dimensions and the 3D data associated with the type of item.

2. The system of claim 1 , wherein the one or more estimated dimensions include height, width, and depth that describe a candidate volume.

3. The system of claim 1 , wherein the working volume comprises a volume bounded at least in part by a portion of the fixture that is designated to store items.

4. The system of claim 1 , wherein the image data is obtained by a camera having a centerline of a field-of-view that is at an oblique angle relative to a plane of the fixture.

5. The system of claim 4 , the computer-executable instructions to determine the one or more estimated item locations associated with the one or more estimated tops of the type of item at the fixture further comprising computer-executable instructions to:
determine four or more points of a top of the one or more estimated tops of the type of item at the fixture;
determine a 3D rotation of the top of the one or more estimated tops of the type of item at the fixture relative to the camera;
determine a 3D translation of the top of the one or more estimated tops of the type of item at the fixture relative to the camera; and
determine four or more points of an estimated item location in 3D space using the 3D rotation and the 3D translation and extrinsic camera parameters of the camera.

6. The system of claim 1 , further comprising computer-executable instructions to:
determine a first estimated item location is above and intersects at least in part a second estimated item location;
remove the second estimated item location from the first set of the one or more estimated item locations; and
extend the first estimated item location down to a surface of the fixture.

7. The system of claim 1 , the computer-executable instructions to determine, using the image data, the one or more estimated tops of the type of item at the fixture further comprising computer-executable instructions to:
access trained model data associated with identification of a corresponding top of a corresponding item; and
process the image data with a classifier using the trained model data to recognize at least one particular top of one particular item using the trained model data.

8. The system of claim 1 , wherein the type of item comprises a rigid container, and the computer-executable instructions to determine the one or more estimated tops of the type of item at the fixture further comprising computer-executable instructions to:
process the image data using a histogram of oriented gradients algorithm to determine one or more occurrences of the type of item in the image data.

9. A system comprising:
a shelf to hold one or more of a type of item;
a camera positioned above the shelf and having a field-of-view that includes at least a portion of the shelf; and
a computing device comprising:
a memory, storing computer-executable instructions; and
a hardware processor to execute the computer-executable instructions to:
acquire image data obtained by the camera;
identify the type of item associated with the shelf;
access item data of the type of item, wherein the item data comprises:
two-dimensional (2D) data indicative of at least a portion of a top of the type of item, and
three-dimensional (3D) data indicative of an overall shape and size of the type of item;

determine one or more tops of the type of item at the shelf using the image data;
determine one or more estimated item locations associated with the one or more tops of the type of item at the shelf;
determine at least one of the one or more estimated item locations within a working volume of the shelf, the working volume comprising a volume bounded by one or more portions of the shelf; and
determine a quantity of the type of item at the shelf using the at least one of the one or more estimated item locations and the 3D data.

10. The system of claim 9 , wherein:
the volume bounded by the one or more portions of the shelf comprises a volume associated with a lane of the shelf within which the type of item may be stowed and not intrude on components of the shelf and adjacent lanes.

11. The system of claim 9 , further comprising computer-executable instructions to:
process the image data with a classifier to recognize the one or more tops of the type of item at the shelf.

12. A method comprising:
accessing image data acquired by a camera of a fixture;
identifying a type of item associated with the fixture;
accessing item data of the type of item, wherein the item data comprises:
two-dimensional (2D) data indicative of at least a portion of a top of the type of item, and
three-dimensional (3D) data associated with the type of item;

determining four or more points in 3D space associated with one or more tops of the type of item;
determining one or more estimated item locations of the one or more tops of the type of item that are within a working volume of the fixture, the working volume comprising a volume within which the type of item may be stowed on a lane of a shelf; and
determining a quantity of the type of item at the fixture based on the one or more estimated item locations.

13. The method of claim 12 , the determining the quantity of the type of item at the fixture further comprising determining an integer number of times that a volume of an individual item as indicated by the 3D data will fit within the one or more estimated item locations.

14. The method of claim 12 , further comprising:
determining a first quantity of the type of item at the fixture using first image data acquired at a first time;
determining a second quantity of the type of item at the fixture using second image data acquired at a second time; and
determining a change in quantity using the first quantity and the second quantity.

15. The method of claim 12 , the determining the four or more points in 3D space associated with the one or more tops of the type of item further comprising:
accessing trained model data associated with identification of the type of item in the image data; and
processing the image data with a machine vision module trained using the trained model data to recognize the one or more tops of the type of item.

16. The method of claim 12 , further comprising:
acquiring, using a second camera, a plurality of images of an item of the type of item, wherein the plu

Assignees

• Amazon Tech Inc

Inventors

• Le Vuong Van
• Tighe Joseph Patrick

Classifications

• G06T2207/20081

  Training; Learning · CPC title

• G06T7/62 Primary

  of area, perimeter, diameter or volume · CPC title

• G06T2207/10012

  Stereo images · CPC title

• G06T2207/10028

  Range image; Depth image; 3D point clouds · CPC title

• G06T2200/04

  involving 3D image data · CPC title