Methods and arrangements for identifying objects

Patent metadata

Field	Value
Publication number	US-10192087-B2
Application number	US-201615175997-A
Country	US
Kind code	B2
Filing date	Jun 7, 2016
Priority date	Aug 30, 2011
Publication date	Jan 29, 2019
Grant date	Jan 29, 2019

Abstract

In some arrangements, product packaging is digitally watermarked over most of its extent to facilitate high-throughput item identification at retail checkouts. Imagery captured by conventional or plenoptic cameras can be processed (e.g., by GPUs) to derive several different perspective-transformed views—further minimizing the need to manually reposition items for identification. Crinkles and other deformations in product packaging can be optically sensed, allowing such surfaces to be virtually flattened to aid identification. Piles of items can be 3D-modelled and virtually segmented into geometric primitives to aid identification, and to discover locations of obscured items. Other data (e.g., including data from sensors in aisles, shelves and carts, and gaze tracking for clues about visual saliency) can be used in assessing identification hypotheses about an item. A great variety of other features and arrangements are also detailed.

First claim

We claim:

1. An image processing method to identify a canned, barcoded item selected for purchase by a first shopper at a retail store, the store having a shopping portion where items are stocked for sale, and a checkout portion, the method including using information from a first sensor, and then using information from second sensor different than the first sensor, in a data fusion manner to narrow a universe of possible item identifications, the method more particularly comprising the acts:
based on information from the first sensor, developing a set of plural item identification hypotheses about the canned barcoded item selected by the shopper for purchase, each of said item identification hypotheses having an associated confidence score that respectively indicates certainty about said hypothesis;
refining the set of plural identification hypotheses about the selected barcoded item, by applying data fusion based on other information, the other information including information from the second sensor, said refining including revising at least certain of said associated confidence scores;
if need be, successively repeating said refining act, until one of said revised confidence scores has a value exceeding a particular, predetermined threshold value, said threshold value having been established prior to said refining act, and serving as a basis for comparison in judging the revised confidence scores; and
adding, to a purchase tally for said shopper, an item associated with said one revised confidence score that has a value exceeding the threshold value, said item being the canned barcoded item selected for purchase by the shopper;
wherein one of said sensors is a fixed sensor in an aisle of the shopping portion of the store, the information from said fixed sensor that is used in said data fusion manner more particularly comprising (a) information indicating locations visited by the shopper during a track of the shopper through the store, or (b) information indicating both removal of an object from a store shelf location and a time of such removal; and
the other of said sensors is a camera at the checkout portion of the retail store, the information from said camera that is used in said data fusion manner more particularly-comprising imagery depicting the canned, barcoded item, or a person's interaction with said item.

2. The method of claim 1 in which the information from the fixed sensor comprises information indicating locations visited by the shopper during a track of the shopper through the store.

3. The method of claim 2 in which the data fusion includes revising at least certain of said confidence scores based on information about a layout of the store, indicating locations of different items in the store.

4. The method of claim 1 in which the information from the fixed sensor comprises information from a ceiling-, floor- or shelf-mounted camera.

5. The method of claim 1 in which the data fusion includes revising at least certain of said confidence scores based on information identifying a store aisle through which the shopper did not pass.

6. The method of claim 1 in which the data fusion includes revising at least certain of said confidence scores based on information about store locations at which the shopper, or the shopper's cart, paused during the shopper's track through the store.

7. The method of claim 1 in which the data fusion includes revising at least certain of said confidence scores based on information about a store location at which an item was placed into the shopper's cart.

8. The method of claim 1 in which the information from the camera at the checkout portion of the store comprises information from a camera at an item bagging area of the store.

9. The method of claim 1 in which the information from the fixed sensor comprises information indicating both removal of an object from the store shelf location, and a time of such removal.

10. The method of claim 9 that includes sensing removal of stock from the store shelf location with a weight transducer or a camera.

11. The method of claim 10 that includes sensing removal of stock from the store shelf location with a camera, and sensing information about locations visited by the shopper with a camera.

12. The method of claim 9 wherein a weight given to the information from the fixed sensor in determining a confidence score depends, at least in part, on the time data.

13. The method of claim 1 in which the data fusion includes revising at least certain of said confidence scores based on information derived from one of the following: sensing disruption of air blown through a pile of items, sensing temperature, sensing vibration, sensing sound, ultrasonic or millimeter wave scanning, sensing ultraviolet or infrared radiation, weight sensing, haptic sensing, 3D shape sensing, chemical or olfactory sensing, inertial modeling, statistical item correlations, or crowdsourced human assessment of item imagery.

14. The method of claim 1 that further includes:
determining an item of information relating to the shopper from the list consisting of: sensed shopper demographic, sensed shopper gaze, sensed pose of a shopper hand grasping an item, shopper purchasing history, and shopper shopping list; and
revising at least one of said confidence scores based on said determined item of information.

15. The method of claim 1 that further includes:
determining one of the following types of information: volumetric product shape information, color histogram data, incomplete or ambiguous barcode data, digital watermark data, NFC/RFID information, image fingerprint data, recognized text data, weight information, temperature information, and information about relative placement of different items in a bag; and
revising at least one of said confidence scores based on said determined item of information.

16. The method of claim 1 wherein both said first shopper and a further shopper both select a second item for purchase, the method including:
attempting to identify the second item selected for purchase by the first shopper, but failing to do so because the continued refinement of the set of plural identification hypotheses does not yield any revised confidence score having a value exceeding the threshold value;
flagging the second item, selected for purchase by the first shopper, to a human operator for follow-up;
attempting to identify the second item selected for purchase by the second shopper, but failing to do so because the continued refinement of the set of plural identification hypotheses does not yield any revised confidence score having a value exceeding the threshold value; and
enabling the second shopper to self-identify the second item;
wherein the second shopper, but not the first shopper, is permitted to self-identify the second item, because the second shopper is determined to have a heightened level of trust with the store.

17. The method of claim 1 that includes sensing, from imagery captured by the camera at the checkout portion of the retail store: image fingerprint data for the item, color histogram data for the item, optical character recognition data for the item, volumetric product configuration data for the item, incomplete or ambiguous barcode or watermark data for the item, barcode position information for the item, barcode aspect ratio for the item, information indicating an order in which items are placed into a bag, information indicating a person's hand pose as the item is grasped, or information indicating a person's gaze towards the item; and
employing said sensed information i

Assignees

• Digimarc Corp

Inventors

• Davis Bruce L
• Rodriguez Tony F
• Rhoads Geoffrey B
• Lord John D
• Evans Eric D
• Gerlach Rebecca L
• Bai Yang
• Stach John

Classifications

• G06K7/10861 Primary

  sensing of data fields affixed to objects or articles, e.g. coded labels (postal sorting B07C3/14, conveying articles B65G47/48) · CPC title

• G06F3/147 Primary

  using display panels · CPC title

• G06V10/464

  using a plurality of salient features, e.g. bag-of-words [BoW] representations · CPC title

• G06V30/2247

  Characters composed of bars, e.g. CMC-7 · CPC title

• G06V10/242

  by image rotation, e.g. by 90 degrees · CPC title