Image rectification using transformation data

Patent metadata

Field	Value
Publication number	US-10909667-B1
Application number	US-201816222795-A
Country	US
Kind code	B1
Filing date	Dec 17, 2018
Priority date	Sep 28, 2015
Publication date	Feb 2, 2021
Grant date	Feb 2, 2021

Abstract

Image data representative of an inventory location may be acquired by cameras. Described are techniques to process such image data to remove one or more perspective effects. For example, transformation data that corresponds to the inventory location may first be determined. Such transformation data is determined based on associations between points in a common plane and alternate points in a virtual camera plane. Transformed image data may then be generated by applying the transformation data to the image data that is representative of the inventory location.

First claim

What is claimed is:

1. A system comprising:
a shelf to hold a plurality of a type of item;
a camera positioned above the shelf and having a field-of-view that includes at least a portion of the shelf; and
a computing device comprising:
a memory, storing computer-executable instructions; and
a hardware processor to execute the computer-executable instructions to:
access image data representative of a foreshortened image of tops of the plurality of the type of item at the shelf;
determine transformation data associated with an inventory location, the transformation data associating a plurality of reference points in a common plane and a plurality of alternate points in a virtual camera plane;
generate first transformed image data from the image data using the transformation data, the first transformed image data representing the tops of the plurality of the type of item with similar apparent sizes;
identify the plurality of the type of item in the first transformed image data; and
determine item count data indicative of a quantity of the plurality of the type of item in the first transformed image data.

2. The system of claim 1 , the hardware processor to further execute the computer-executable instructions to:
calculate the transformation data based on one or more of:
size information associated with physical dimensions of the shelf, layout information associated with a layout of the shelf and the camera,
parameter information associated with one or more parameters of the camera,
configuration information associated with a configuration of the camera, or
calibration data associated with the plurality of the type of item.

3. The system of claim 1 , the hardware processor to further execute the computer-executable instructions to:
access item data associated with the plurality of the type of item;
determine at least one height associated with the plurality of the type of item; and
wherein the determination of the transformation data is based on calibration data that is related to the at least one height associated with the plurality of the type of item.

4. A system comprising:
a memory, storing computer-executable instructions; and
a hardware processor to execute the computer-executable instructions to:
access image data representative of a foreshortened image of a plurality of a type of item at an inventory location;
determine transformation data associated with the inventory location, the transformation data associating a plurality of reference points in a common plane and a plurality of alternate points in a virtual camera plane;
generate first transformed image data from the image data using the transformation data, the first transformed image data representing the plurality of the type of item with sizes that are similar within a given threshold; and
generate first output data using the first transformed image data.

5. The system of claim 4 , the hardware processor to further execute the computer-executable instructions to:
calculate the transformation data based on facility information or calibration data.

6. The system of claim 5 , wherein the facility information comprises at least one or more of:
size information associated with one or more physical dimensions of the inventory location,
layout information associated with a layout of at least one or more of a shelf and one or more sensors at the inventory location,
parameter information related to one or more parameters of the one or more sensors, or
configuration information associated with the one or more sensors; and
wherein the calibration data is associated with at least the plurality of the type of item at the inventory location.

7. The system of claim 4 , the hardware processor to further execute the computer-executable instructions to:
calculate the transformation data based on a planar target temporarily placed at a known height above the inventory location; and
wherein the transformation data is determined based on the known height corresponding to a height of the plurality of the type of item at the inventory location.

8. The system of claim 4 , the hardware processor to further execute the computer-executable instructions to:
generate first calibration data for a first known height above the inventory location;
generate second calibration data for a second known height above the inventory location; and
wherein the transformation data is determined based at least on one or more of:
the first calibration data, or
the second calibration data.

9. The system of claim 4 , wherein:
the first output data comprises one or more of:
item count data, the item count data comprising information indicative of a quantity of the plurality of the type of item at the inventory location, or
item identification data, the item identification data comprising information indicative of the plurality of the type of item at the inventory location.

10. The system of claim 4 , the hardware processor to further execute the computer-executable instructions to:
determine a count of the plurality of the type of item in the first transformed image data using a convolutional neural network that detects individual items in the first transformed image data; and
wherein the generation of the first output data uses the count of the plurality of the type of item.

11. The system of claim 4 , the hardware processor to further execute the computer-executable instructions to:
determine one or more features of the plurality of the type of item in the first transformed image data using a histogram of oriented gradients algorithm;
classify the one or more features of the plurality of the type of item using a support vector machine;
determine a count of the plurality of the type of item in the first transformed image data; and
wherein the generation of the first output data uses the count of the plurality of the type of item.

12. The system of claim 4 , the hardware processor to further execute the comprising computer-executable instructions to:
generate interaction data at the inventory location using the first output data.

13. A method comprising:
accessing image data representative of a plurality of a type of item at an inventory location;
identifying the plurality of the type of item at the inventory location;
determining transformation data associated with the inventory location, the transformation data associating a plurality of reference points in a common plane and a plurality of alternate points in a virtual camera plane;
generating first transformed image data from the image data using the transformation data, the first transformed image data representing the plurality of the type of item with sizes that vary less than a given threshold; and
generating first output data using the first transformed image data.

14. The method of claim 13 , further comprising:
calculating the transformation data using one or more of:
size information associated with one or more physical dimensions of a shelf at the inventory location,
layout information associated with a layout of at least one or more of the shelf or one or more sensors at the inventory location,
parameter information related to one or more parameters of the one or more sensors,
configuration information associated with the one or more sensors, or
calibration data associated with the plurality of the type of item at the inventory location.

15. The method of claim 13 , further comprising:
calculating the transformation data based on a planar target temporarily placed at a known height above the inventory location; and
wherein a height

Assignees

• Amazon Tech Inc

Inventors

• Tighe Joseph Patrick
• Le Vuong Van
• Medioni Gerard Guy

Classifications

• G06V10/247

  by affine transforms, e.g. correction due to perspective effects; Quadrilaterals, e.g. trapezoids · CPC title

• G06V20/52

  Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69) · CPC title

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title

• G06T5/40

  using histogram techniques · CPC title

• G06T2207/30232

  Surveillance · CPC title