Reducing the search space for recognition of objects in an image based on wireless signals

Patent metadata

Field	Value
Publication number	US-9684826-B2
Application number	US-201514839058-A
Country	US
Kind code	B2
Filing date	Aug 28, 2015
Priority date	Aug 28, 2014
Publication date	Jun 20, 2017
Grant date	Jun 20, 2017

Abstract

Provided is a process including: determining that a mobile computing device has crossed a geofence associated with a merchant store; sending, to a remote classifier server, a request for object-recognition classifiers for objects in the merchant store; receiving a set of object-recognition classifiers; receiving with the mobile computing device from user a request to search for offers; capturing an image with a camera of the mobile computing device; receiving one or more wireless beacon identifiers with the mobile computing device; based on the wireless beacon identifiers, selecting a subset of the object-recognition classifiers in the set of object-recognition classifiers; and recognizing an object in the captured image based on the selected subset of the object-recognition classifiers; and requesting, from a remote offer publisher server, offers corresponding to the recognized object; and receiving offers from the remote offer publisher server; and displaying the received offers to the user.

First claim

What is claimed is:

1. A method of searching for electronic coupons or other offers based on an image captured by a camera of a mobile user device of a product for sale in a merchant's store, wherein the product is recognized in the image by the mobile user device without sending the image to a remote computer for image recognition, the method comprising:
obtaining, in memory of a mobile user device, a set of object-recognition classifiers, the object-recognition classifiers each specifying one or more algorithms and parameters of those algorithms by which respective products among a plurality of products in a merchant store are recognized in images;
commanding, with one or more processors of the mobile user device, a camera of the mobile user device to capture an image;
receiving the captured image;
receiving, via an antenna of the mobile user device, a wirelessly transmitted beacon identifier that indicates a location of the mobile user device within the merchant store, wherein indicating the location of the mobile user device includes distinguishing the location of the mobile user device from a plurality of other locations in the merchant store where other wireless beacon identifiers are transmitted;
selecting a subset of the object-recognition classifiers based on the received beacon identifier, the subset of the object-recognition classifiers being configured to indicate whether at least some products positioned within range of the received wireless beacon identifier appear in the image;
recognizing a product in the image by executing the algorithms specified by the subset of object-recognition classifiers with the one or more processors of the mobile user device; and
obtaining a coupon or other offer based on the recognized product.

2. The method of claim 1 , wherein:
obtaining the set of object-recognition classifiers comprises, before commanding the camera of the mobile user device to capture the image:
determining that a mobile user device has crossed a geofence associated with the merchant store based on cellular or satellite navigation signals received by the mobile user device while outside of the merchant store, beacons received by the mobile computer device, or a combination thereof;
sending, to a remote classifier server, a request for object-recognition classifiers for products in the merchant store; and
receiving, from the remote classifier server, a set of object-recognition classifiers;

at least one of the object-recognition classifiers comprises:
an identifier of a given product;
identifiers of a plurality of object-recognition functions stored in the memory of the mobile user device before obtaining the set of object-recognition classifiers;
weightings and a sequence that specify a cascaded arrangement of the object-recognition functions; and
parameters of the object-recognition functions that cause the image recognition function to output a value indicative of whether the given product appears in the image;

receiving the captured image comprises receiving a plurality of video frames captured over a trailing duration;
receiving the wirelessly transmitted beacon identifier comprises receiving a Low-Energy Bluetooth beacon and measuring a beacon signal strength that together specify the location of the mobile user device in an aisle of the merchant store to within less than plus-or-minus five meters;
selecting the subset of the object-recognition classifiers based on the received beacon identifier comprises:
determining a product section of the merchant store based on the received beacon identifier; and
filtering from the set of object-recognition classifiers those classifiers that identify a product in the product section of the merchant store to exclude those object-recognition classifiers that do not pertain to the product section of the merchant store;

recognizing the product in the image comprises:
for each video frame among the plurality of video frames, calculating a score with each classifier among the subset of the object-recognition classifiers;
calculating a measure of central tendency among the plurality of video frames for each classifier among the subset of the object-recognition classifiers; and
determining that the measure of central tendency of the classifier corresponding to the product satisfies a threshold;

obtaining the coupon or other offer based on the recognized product comprises:
parsing from the classifier corresponding to the product a text string;
sending a query including the text string to a remote offers engine;
receiving one or more offers responsive to the query; and
presenting at least some of the one or more offers on a display screen of the mobile user device.

3. The method of claim 1 , wherein the wirelessly transmitted beacon identifier is a Low-Energy Bluetooth beacon identifier transmitted by one of a plurality of battery-powered Low-Energy Bluetooth beacons distributed in the merchant store, the method further comprising:
measuring a signal strength of the signal with which the wirelessly transmitted beacon identifier is received; and
determining a location of the mobile user device in the merchant store based on both the signal strength and the received beacon identifier.

4. The method of claim 1 , wherein selecting a subset of the object-recognition classifiers based on the received beacon identifier comprises:
determining a product section of the merchant store in which the mobile user device is located and filtering from the set of object-recognition classifiers those classifiers associated with a product in the product section of the merchant store.

5. The method of claim 1 , wherein obtaining, in memory of the mobile user device, the set of object-recognition classifiers comprises:
obtaining one or more cascades of classifiers, each stage of the cascades corresponding to a different classifier from either an upstream classifier or a downstream classifier in the respective cascade, and each cascade of classifiers being configured to output a score indicative of whether a given product is depicted in a given image.

6. The method of claim 1 , wherein
receiving the captured image comprises storing a plurality of video frames with the camera of the mobile device in a video buffer; and
recognizing a product in the image comprises:
calculating scores with each of the subset of object-recognition classifiers for each of the frames of video in the video buffer;
calculating a measure of central tendency of the scores for each of the classifiers among the subset of object-recognition classifiers; and
determining whether a product is present in the video frames by determining whether at least one of the measures of central tendency satisfies a threshold.

7. The method of claim 1 , comprising:
polling an accelerometer of the mobile user device to receive a value indicative of movement of the mobile user device; and
determining to recognize the product in the image in response to the value indicating less than a threshold amount of movement.

8. The method of claim 1 , wherein at least some of the object-recognition classifiers each comprise:
an identifier of a given product;
an identifier of an object-recognition function stored in the memory of the mobile user device before obtaining the set of object-recognition classifiers; and
parameters of the object-recognition function that cause the object-recognition function to output a value indicative of whether the given product appears in an image.

9. The method of claim 1 , wherein obtaining the set of object-recognition classifiers comprises:
determining that the mobile user device is in a geographic area associated with the merchant store and, in response:
sending, t

Assignees

• Retailmenot Inc

Inventors

• Dubuque Shaun F

Classifications

• G06V10/80

  Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level (multimodal speaker identification or verification G10L17/10) · CPC title

• H04W4/02 Primary

  Services making use of location information · CPC title

• G06V10/87

  using selection of the recognition techniques, e.g. of a classifier in a multiple classifier system · CPC title

• G06V10/764

  using classification, e.g. of video objects · CPC title

• G06F2218/12

  Classification; Matching · CPC title