Sensor-based mobile search, related methods and systems

Patent metadata

Field	Value
Publication number	US-9557162-B2
Application number	US-201313943176-A
Country	US
Kind code	B2
Filing date	Jul 16, 2013
Priority date	Oct 28, 2009
Publication date	Jan 31, 2017
Grant date	Jan 31, 2017

Abstract

A smart phone senses audio, imagery, and/or other stimulus from a user's environment, and acts autonomously to fulfill inferred or anticipated user desires. In one aspect, the detailed technology concerns phone-based cognition of a scene viewed by the phone's camera. The image processing tasks applied to the scene can be selected from among various alternatives by reference to resource costs, resource constraints, other stimulus information (e.g., audio), task substitutability, etc. The phone can apply more or less resources to an image processing task depending on how successfully the task is proceeding, or based on the user's apparent interest in the task. In some arrangements, data may be referred to the cloud for analysis, or for gleaning. Cognition, and identification of appropriate device response(s), can be aided by collateral information, such as context. A great number of other features and arrangements are also detailed.

First claim

We claim:

1. A method comprising:
receiving first data corresponding to imagery captured by a camera of a smartphone, the imagery depicting a visual subject;
deriving recognition features from the imagery, said deriving being performed by a processing system in the smartphone configured to perform such act;
receiving second data corresponding to non-image stimulus captured by a sensor of the smartphone, said non-image stimulus comprising at least one stimulus selected from the group consisting of: audio, temperature, magnetic field, smell, or chemical presence;
from a set of reference recognition features associated with a first set of visual subjects, identifying a smaller subset of recognition features associated with a second, smaller set of visual subjects, said smaller set of visual subjects including first and second visual subjects;
using the non-image stimulus, classifying an environment of the smartphone by assigning a first probability value that the smartphone environment is a first environment and assigning a second probability value that the smartphone environment is a second environment, both of said probability values being more than 0% and less than 100%;
obtaining two values respectively indicating likelihoods of encountering the first visual subject in said first and second environments, and obtaining two other values respectively indicating likelihoods of encountering the second visual subject in said first and second environments; and
combining said probability and likelihood values together in assessing that the visual subject is more likely to be the first visual subject than the second visual subject:
wherein the visual subject is identified from among said second set of subjects, by correspondence between the derived recognition features and recognition features in said subset, and by use of said probability and likelihood values.

2. The method of claim 1 wherein said recognition features comprise SIFT features.

3. The method of claim 1 wherein said non-image stimulus comprises non-speech audio.

4. The method of claim 1 wherein said non-image stimulus comprises audio from a source different than said subject.

5. The method of claim 1 wherein said non-image stimulus comprises temperature.

6. The method of claim 1 wherein said non-image stimulus comprises magnetic field.

7. The method of claim 1 wherein said non-image stimulus comprises smell.

8. The method of claim 1 wherein said non-image stimulus comprises chemical presence.

9. The method of claim 1 that includes identifying the smaller subset of recognition features associated with the second, smaller set of visual subjects based, at least in part, on the second data.

10. A non-transitory computer readable medium containing software instructions operative to configure a processor- and camera-equipped smartphone system to perform acts including:
receiving first data corresponding to imagery captured by the camera, the imagery depicting a visual subject;
deriving recognition features from the imagery;
receiving second data corresponding to non-image stimulus captured by a sensor of the smartphone system, said non-image stimulus comprising at least one stimulus selected from the group consisting of: audio, temperature, magnetic field, smell, or chemical presence;
from a set of reference recognition features associated with a first set of visual subjects, identifying a smaller subset of recognition features associated with a second, smaller set of visual subjects, said smaller set of visual subjects including first and second visual subjects;
using the non-image stimulus, classifying an environment of the camera by assigning a first probability value that the camera environment is a first environment and assigning a second probability value that the camera environment is a second environment, both of said probability values being more than 0% and less than 100%;
obtaining two values respectively indicating likelihoods of encountering the first visual subject in said first and second environments, and obtaining two other values respectively indicating likelihoods of encountering the second visual subject in said first and second environments; and
combining said probability and likelihood values together in assessing that the visual subject is more likely to be the first visual subject than the second visual subject;
wherein the visual subject is identified from among said second set of subjects, by correspondence between the derived recognition features and recognition features in said subset, and by use of said probability and likelihood values.

11. A smartphone system including the computer readable memory of claim 10 , together with one or more processors, a screen, a touch sensor, a camera, and a wireless interface.

Assignees

• Digimarc Corp

Inventors

• Rodriguez Tony F
• Conwell William Y

Classifications

• G06V10/765

  using rules for classification or partitioning the feature space · CPC title

• H04W4/02 Primary

  Services making use of location information · CPC title

• G06F2218/08

  Feature extraction · CPC title

• G01B11/14 Primary

  for measuring distance or clearance between spaced objects or spaced apertures (G01B11/26 takes precedence; rangefinders G01C3/00) · CPC title

• G06F18/24765

  Rule-based classification · CPC title