Fusion of sensor data for detecting interactions at an inventory location

Patent metadata

Field	Value
Publication number	US-11042836-B1
Application number	US-201916435268-A
Country	US
Kind code	B1
Filing date	Jun 7, 2019
Priority date	Jun 7, 2019
Publication date	Jun 22, 2021
Grant date	Jun 22, 2021

Abstract

Items may be stowed in an inventory location, such as a shelf. An interaction may take place where one of the items is picked from or placed on the inventory location. Sensor data can be acquired from two or more sensors, such as cameras or weight sensors, configured to collect such sensor data for the inventory location. Hypotheses to describe interactions at the inventory location may be determined using sensor data from each different sensor. However, the confidence value for such hypotheses may not be reliable. Fusion of sensor data may be performed where sets of hypotheses from the different sensors are combined and the resulting hypotheses are more accurate.

First claim

What is claimed is:

1. A system comprising:
a shelf to hold one or more items;
a camera with a field-of-view comprising at least a portion of the shelf;
a sensor to generate sensor data for the shelf; and
a computing device comprising:
a memory storing computer-executable instructions; and
a hardware processor to execute the computer-executable instructions to:
access image data acquired by the camera;
determine an interaction has occurred at the shelf involving the one or more items based on the image data;
determine a start time and an end time of the interaction;
determine, based on at least a portion of the image data, a first set of hypotheses for the interaction, each hypothesis of the first set of hypotheses comprising:
one or more first item identifiers, and
a first probability value that the one or more first item identifiers were involved in the interaction;

access the sensor data generated by the sensor, the sensor data comprising non-image data;
determine, based on the sensor data, a second set of hypotheses for the interaction, each hypothesis of the second set of hypotheses comprising:
one or more second item identifiers, and
a second probability value that the one or more second item identifiers were involved in the interaction;

determine that the first set of hypotheses and the second set of hypotheses do not meet or exceed a confidence threshold value;
generate a third set of hypotheses for the interaction by combining the first set of hypotheses and the second set of hypotheses;
determine at least one hypothesis of the third set of hypotheses has a confidence value that meets or exceeds the confidence threshold value;
select a hypothesis from the third set of hypotheses as a solution to describe the interaction; and
update inventory quantities for the one or more items on the shelf using the solution.

2. The system of claim 1 , the hardware processor to further execute the computer-executable instructions to:
access item data;
access a first image of the shelf at the start time of the interaction;
access a second image of the shelf at the end time of the interaction;
determine a first count of items at the start time of the interaction based on the first image and the item data;
determine a second count of items at the end time of the interaction based on the second image and the item data; and
determine a quantity of items added to or removed from the shelf by subtracting the first count of the items from the second count of the items.

3. The system of claim 1 , wherein:
the sensor comprises a weight sensor;
the sensor data comprises weight data; and
the third set of hypotheses is generated using Bayes' rule.

4. A system comprising:
a computing device comprising:
a memory storing computer-executable instructions; and
a hardware processor to execute the computer-executable instructions to:
access image data acquired by a camera with a field-of view inclusive of an inventory location, the inventory location stowing one or more items;
determine an interaction involving one or more of the one or more items at the inventory location has occurred, based on one or more images of the image data;
determine, based on at least a portion of the image data, a first set of hypotheses for the interaction, each hypothesis of the first set of hypotheses indicative of a first probability that one or more first items were involved in the interaction;
access sensor data acquired by one or more sensors, the one or more sensors configured to generate the sensor data for the inventory location;
determine, based on the sensor data, a second set of hypotheses for the interaction, each hypothesis of the second set of hypotheses indicative of a second probability that one or more second items were involved in the interaction;
determine that the first set of hypotheses and the second set of hypotheses do not meet or exceed a confidence threshold value;
generate a third set of hypotheses for the interaction by combining at least a portion of the first set of hypotheses and at least a portion of the second set of hypotheses;
determine that at least one hypothesis in the third set of hypotheses has a probability value that meets or exceeds the confidence threshold value;
select a hypothesis from the third set of hypotheses as a solution that describes the interaction; and
update inventory quantities of one or more of the one or more items stowed at the inventory location based on the solution.

5. The system of claim 4 , the hardware processor to further execute the computer-executable instructions to:
access a first image of the image data acquired before the interaction occurs;
access a second image of the image data acquired after the interaction occurs; and
determine differential data indicative of a change at the inventory location using the first image and the second image.

6. The system of claim 5 , wherein the computer-executable instructions to determine the differential data indicative of the change at the inventory location further comprise instructions to:
determine the change at the inventory location occurs at a partitioned area of the inventory location; and
determine one or more of addition or removal of items from the partitioned area of the inventory location.

7. The system of claim 5 , wherein the computer-executable instructions to determine the differential data indicative of the change at the inventory location further comprise instructions to:
access item data;
determine a starting count of items appearing in the first image;
determine an ending count of items appearing in the second image; and
determine a quantity of items added to or removed from the inventory location by subtracting the starting count from the ending count.

8. The system of claim 5 , wherein the computer-executable instructions to determine the differential data indicative of the change at the inventory location further comprise instructions to:
detect first indicia or markings at the inventory location at a beginning of the interaction using the first image, wherein the first indicia or markings comprise one or more of:
tracking marks,
bar codes, or
a ruler printed on a surface of the inventory location;

detect second indicia or markings at the inventory location at an end of the interaction using the second image; and
compare the first indicia or markings and the second indicia or markings to determine whether items have been added to or removed from the inventory location.

9. The system of claim 4 , the hardware processor to further execute the computer-executable instructions to:
determine the interaction involves a change to at least one of a plurality of partitioned areas of the inventory location;
access item data to determine items stowed at the at least one of the plurality of partitioned areas of the inventory location; and
wherein the first set of hypotheses is determined using the at least a portion of the image data and the item data.

10. The system of claim 4 , wherein:
the third set of hypotheses is generated using Bayes' rule.

11. The system of claim 4 , the hardware processor to further execute the computer-executable instructions to:
determine, based on the solution, interaction data indicative of one or more of:
a change in quantity of the one or more of the one or more items at the inventory location,
an item identifier indicative of the one or more of the one or more items involved in the interaction, or
at least one partitioned area of the inventory location from which the one or more of the one or more items were picked or placed.

12. The system of claim 4 ,

Assignees

• Amazon Tech Inc

Inventors

• Goldstein Michel Leonard
• Palaniappan Ramanathan
• Sun Fan
• Bo Liefeng
• Manyam Ohil Krishnamurthy
• Shiee Navid
• Medioni Gerard Guy

Classifications

• G07G1/0036

  Checkout procedures · CPC title

• G07F9/026

  for alarm, monitoring and auditing in vending machines or means for indication, e.g. when empty · CPC title

• G06Q20/208

  Input by product or record sensing, e.g. weighing or scanner processing · CPC title

• G06Q20/203

  Inventory monitoring · CPC title

• G06Q10/00

  Administration; Management · CPC title