Intuitive computing methods and systems

Patent metadata

Field	Value
Publication number	US-2016379082-A1
Application number	US-201615259882-A
Country	US
Kind code	A1
Filing date	Sep 8, 2016
Priority date	Oct 28, 2009
Publication date	Dec 29, 2016
Grant date	—

Abstract

In one particular aspect, a portable computing device (e.g., a tablet or smartphone) senses audio and/or image content from a user's environment, and initiates one or more recognition agents (e.g., performing image watermark recognition, image recognition, object recognition, facial recognition, barcode recognition, optical character recognition, audio watermark recognition, speech recognition, speaker recognition, or music recognition). Resource allocation to a recognition agent can be varied based on (a) progress of the recognition agent to achieve its recognition goal, and (b) user interest data indicating user interest in the output of the recognition agent. A second candidate recognition agent can be evaluated for possible launch, based on a relevance score, and a cost score. In some embodiments, the device adapts its operation to changing context, by terminating a first recognition agent in favor of a second recognition agent, without express user instruction to do so. A great number of other features and arrangements are also detailed.

First claim

1 . A method practiced by a battery-powered mobile wireless communications device equipped with a camera and microphone, the device having multiple recognition modes for recognizing multiple types of content, the method comprising the acts:
initiating one or more stages of a first recognition agent process to recognize audio or image content data captured by the mobile device, the first recognition agent process performing a recognition selected from a list consisting of: image watermark recognition, image recognition, object recognition, facial recognition, barcode recognition, optical character recognition, audio watermark recognition, speech recognition, speaker recognition, or music recognition; receiving detection state data indicating a state of the first recognition agent process in performing the selected recognition; after initiating the first recognition agent process, receiving first user interest data indicating interest of the user in obtaining a result of the first recognition agent process; and varying an allocation of processing resources to the first recognition agent process based on both said detection state data and on said user interest data; the method thereby optimizing use of the device battery by allocating processing resources in response both to the state of the first recognition agent process in performing the selected recognition, and to user input received after the first recognition agent process has been initiated, indicating user interest in the result of the first recognition agent process.

2 . The method of claim 1 that further includes:
determining a relevance score for a second recognition agent process that the mobile device may run, the relevance score being based on data including at least one data selected from a list consisting of: location, sensor data available in a memory, context, expressed user intent, and user history;
determining a cost score for said second recognition agent process, the cost score being based on data including at least one data selected from a list consisting of: memory usage, CPU usage, and communication bandwidth; and
based on at least (a) the detection state data for the first recognition agent process, (b) the relevance score of the second recognition agent process, and (c) the cost score for the second recognition agent process, terminating the first recognition agent process and initiating the second agent process;
wherein the device adapts its operation to changing context, by terminating the first recognition agent process in favor of the second recognition agent process, without express user instruction to do so.

3 . The method of claim 1 that further includes:
determining a relevance score for each of plural candidate recognition agent processes that the mobile device may run, the relevance score for a first of said candidate recognition agent processes being based on data including at least one data selected from the list consisting of: location, sensor data available in a memory, context, expressed user intent, and user history;
determining a cost score for each of said plural candidate recognition agent processes, the cost score for a first of said candidate recognition agent processes being based on data including at least one data selected from the list consisting of: memory usage, CPU usage, and communication bandwidth; and
determining, using both the relevance and cost scores, a further recognition agent process to initiate.

4 . The method of claim 1 in which the act of varying the allocation of processing resources to the first recognition agent process depends on a derivative of said detection state data.

5 . The method of claim 1 in which the act of varying the allocation of processing resources to the first recognition agent process depends on a speed or acceleration at which the first recognition agent process is progressing in performing the selected recognition.

6 . The method of claim 1 , performed first and second times, wherein the first time the method is performed, the first recognition agent process is terminated when the detection state data has a first value, and the second time the method is performed, the first recognition agent process is not terminated when the detection state data has said first value, because the received first user interest data indicates a higher level of user interest the second time the method is performed, than the first time the method is performed.

7 . The method of claim 1 in which the received first user interest data comprises data related to express user encouragement of the first recognition agent process.

8 . The method of claim 1 in which the received first user interest data comprises data related to implied user encouragement of the first recognition agent process.

9 . The method of claim 8 in which the implied user encouragement comprises a pose at which the user positions the mobile device.

10 . The method of claim 9 in which the implied user encouragement comprises an orientation at which the user positions the mobile device.

11 . The method of claim 1 in which the first user interest data comprises accelerometer data.

12 . The method of claim 1 in which the first user interest data comprises data relating to a camera zoom function.

13 . The method of claim 1 in which the first user interest data comprises user positioning of the device to place a subject of the first recognition agent process at a position remote from a center of a camera field of view.

14 . The method of claim 1 that further includes:
initiating one or more stages of a second recognition agent process to recognize audio or image data captured by the mobile device, the second recognition agent process being different than the first recognition agent process, the second recognition agent process performing a second recognition selected from the list consisting of: image watermark recognition, image recognition, object recognition, facial recognition, barcode recognition, optical character recognition, audio watermark recognition, speech recognition, speaker recognition, or music recognition;
receiving second detection state data indicating a state of the second recognition agent process in performing the selected second recognition;
after initiating the second recognition agent process, receiving second user interest data indicating interest of the user in obtaining a result of the second recognition agent process; and
varying an allocation of processing resources to the second recognition agent process based on both said second detection state data and on said second user interest data;
wherein the first and second recognition agent processes are performed concurrently.

15 . The method of claim 1 in which the device includes a touch screen display and the method also includes:
initiating a discovery mode, in response to a request received from the user via a device input, in which the discovery mode includes processing camera-captured imagery to recognize an object depicted therein using said first recognition agent process;
presenting captured imagery, including video, in a first area of said display of said device, along with augmented reality graphics;
controlling a user interface to display user-selectable graphic icons in a second area of said display, wherein the user-selectable graphic icons include visible indicia to graphically represent content;
controlling the user interface to present a user-selectable graphic icon on the display to facilitate switching discovery modes from an image discovery mode to an audio discovery mode; and
wherein the audio discover

Assignees

• Digimarc Corp

Inventors

• Rodriguez Tony F
• Rhoads Geoffrey B

Classifications

• H04W4/02 Primary

  Services making use of location information · CPC title

• H04M1/724

  User interfaces specially adapted for cordless or mobile telephones · CPC title

• H04N23/667

  Camera operation mode switching, e.g. between still and video, sport and normal or high- and low-resolution modes · CPC title

• G06F18/24

  Classification techniques · CPC title

• H04N23/80

  Camera processing pipelines; Components thereof · CPC title