Multiple sensor data fusion system

Patent metadata

Field	Value
Publication number	US-10318917-B1
Application number	US-201514675167-A
Country	US
Kind code	B1
Filing date	Mar 31, 2015
Priority date	Mar 31, 2015
Publication date	Jun 11, 2019
Grant date	Jun 11, 2019

Abstract

An inventory location such as a shelf may be used to stow different types of items. Interactions may take place, such as the pick or place of one or more items from the inventory location. Image data may be acquired from cameras viewing the shelf and weight data may be acquired from weight sensors coupled to the shelf. Hypotheses may be determined that indicate possible interactions with the inventory location, such as pick or place of an item with regard to the inventory location, and the probability that those interactions are correct. The hypotheses and their associated probabilities may be aggregated. From the aggregated hypotheses, a hypothesis with a highest confidence value may be deemed a solution.

First claim

What is claimed is:

1. A system comprising:
a shelf to hold a first type of item in a first partitioned area and a second type of item in a second partitioned area;
a plurality of weight sensors, each of the plurality of weight sensors positioned at the shelf at a different point;
a camera with a field-of-view that includes at least a portion of the shelf; and
a computing device comprising:
a memory, storing computer-executable instructions; and
a hardware processor to execute the computer-executable instructions to:
determine that an interaction has been detected involving at least one of the first type of item and the second type of item on the shelf;
responsive to determining that the interaction has been detected, identify that weight data and image data describing the interaction are available, wherein the weight data is acquired by the plurality of weight sensors and the image data is acquired by the camera;
receive the weight data acquired by the plurality of weight sensors;
receive the image data acquired by the camera, wherein the weight data and the image data are acquired contemporaneously with one another;
access item data indicative of item identifiers associated with respective ones of the first partitioned area and the second partitioned area;
generate, from the image data, motion data indicative of an occurrence of motion at the shelf;
determine, from the image data, a start frame acquired before the motion and an end frame acquired after the motion;
generate, from the start frame and the end frame, data indicative of a change in appearance of the shelf;
determine, based on the image data, a first set of hypotheses for the interaction, each hypothesis of the first set including one or more item identifiers with probability values indicative of a probability that the one or more item identifiers of one or more items were involved in the interaction at the shelf;
determine, based on the weight data, a second set of hypotheses for the interaction, each hypothesis of the second set including:
one or more item identifiers,
a predicted quantity of items for each of the one or more item identifiers, and
probability values indicative of a probability that the hypothesis is true;

combine the first set of hypotheses and the second set of hypotheses using Bayes' rule to generate a third set of hypotheses for the interaction, each hypothesis of the third set including:
one or more item identifiers indicative of one or more items; and
predicted quantities of the one or more items with probability values indicative of a probability the hypothesis is true;

determine, from the third set of hypotheses for the interaction, a first hypothesis having a first highest probability and a second hypothesis having a second highest probability;
determine a first confidence value for the first hypothesis based on a difference between the first highest probability and the second highest probability of the third set of hypotheses;
compare the first confidence value and a first confidence value threshold;
determine the first confidence value is below the first confidence value threshold;
determine, responsive to the data indicative of a change in appearance of the shelf and based on the start frame and the end frame, a fourth set of hypotheses for the interaction, each hypothesis of the fourth set including:
one or more predicted quantities of the one or more items with probability values indicative of a probability that the hypothesis is true;

combine the third set of hypotheses for the interaction and the fourth set of hypotheses for the interaction using Bayes' rule to generate a fifth set of hypotheses for the interaction, each hypothesis of the fifth set including:
one or more item identifiers,
the predicted quantity of items for each of the one or more item identifiers, and
probability values indicative of a probability that the one or more item identifiers and the predicted quantity of items for each of the one or more item identifiers is true;

select one hypothesis of the fifth set of hypotheses as a solution that describes the interaction; and
update inventory quantities for the at least one of the first type of item and the second type of item using the solution.

2. The system of claim 1 , further comprising computer-executable instructions to:
determine, from the fifth set of hypotheses, a third hypothesis having a first highest probability and a fourth hypothesis having a second highest probability;
determine a second confidence value for the third hypothesis based on a difference between the first highest probability and the second highest probability of the fifth set of hypotheses;
compare the second confidence value and the first confidence value threshold;
determine the second confidence value is at or above the first confidence value threshold; and
designate the third hypothesis as the solution.

3. The system of claim 2 , further comprising computer-executable instructions to:
determine, based on the solution, interaction data indicative of:
a change in quantity of one or more items resulting from the interaction,
an item identifier indicative of the one or more items involved in the interaction, and
one of the first partitioned area or the second partitioned area at which the one or more items were picked or placed.

4. A system comprising:
a computing device comprising:
a memory, storing computer-executable instructions; and
a hardware processor to execute the computer-executable instructions to:
determine that an interaction has been detected involving one or more items at an inventory location;
in response to determining that the interaction has been detected, identify that weight data and image data describing the interaction are available, wherein the weight data is acquired by a plurality of weight sensors at the inventory location and the image data is acquired by a camera viewing the inventory location;
access the weight data acquired by the plurality of weight sensors at the inventory location;
access the image data acquired by the camera viewing the inventory location;
generate, from the image data, motion data indicative of an occurrence of motion at the inventory location;
determine, from the image data, a start frame acquired before the motion and an end frame acquired after the motion;
determine, based on the image data, a first set of hypotheses for the interaction, each hypothesis in the first set indicative of a probability that a particular item identifier was involved in the interaction;
determine, based on the weight data, a second set of hypotheses for the interaction, each hypothesis in the second set indicative of a probability that a particular quantity of a particular item was involved in the interaction;
determine that a probability value for each of the hypotheses in the first set and second set is below a confidence level threshold;
determine a third set of hypotheses for the interaction by combining at least a portion of the first set of hypotheses for the interaction and at least a portion of the second set of hypotheses for the interaction;
compare a probability value for one or more hypotheses in the third set of hypotheses for the interaction with the confidence level threshold;
determine that one of the one or more hypotheses in the third set of hypotheses has a probability value above the confidence level threshold;
determine that the one of the one or more hypotheses in the third set of hypotheses represents a solution indicative of a quantity of the one or more items that changed at the inventory location; and
update inventory quantities for the one or more items at the inventory location based on the solution.

5. The system of claim 4 , further comprising computer-executable instructio

Assignees

• Amazon Tech Inc

Inventors

• Goldstein Michel Leonard
• Palaniappan Ramanathan
• Sun Fan
• Bo Liefeng
• Manyam Ohil Krishnamurthy
• Shiee Navid
• Medioni Gerard Guy

Classifications

• G06Q10/00 Primary

  Administration; Management · CPC title

• G06Q20/208

  Input by product or record sensing, e.g. weighing or scanner processing · CPC title

• G06Q20/203

  Inventory monitoring · CPC title

• G06Q20/00

  Payment architectures, schemes or protocols (apparatus for performing or posting payment transactions G07F7/08, G07F19/00; electronic cash registers G07G1/12) · CPC title

• G06Q20/20

  Point-of-sale [POS] network systems · CPC title