Methods and arrangements employing sensor-equipped smart phones

Patent metadata

Field	Value
Publication number	US-9609117-B2
Application number	US-201514861758-A
Country	US
Kind code	B2
Filing date	Sep 22, 2015
Priority date	Dec 31, 2009
Publication date	Mar 28, 2017
Grant date	Mar 28, 2017

Abstract

The present technology concerns improvements to smart phones and related sensor-equipped systems. Some embodiments involve spoken clues, e.g., by which a user can assist a smart phone in identifying what portion of imagery captured by a smart phone camera should be processed, or identifying what type of image processing should be conducted. Some arrangements include the degradation of captured content information in accordance with privacy rules, which may be location-dependent, or based on the unusualness of the captured content, or responsive to later consultation of the stored content information by the user. A great variety of other features and arrangements are also detailed.

First claim

We claim:

1. A method employing a portable user system having a processor, a display, at least one microphone that captures audio, and at least one image sensor for capturing imagery, the method comprising the acts:
capturing imagery with the image sensor of the portable user system, the captured imagery depicting plural physical subjects within a physical environment of a user, and presenting the captured imagery to the user on the display;
the processor selecting a first of said depicted plural physical subjects as being of likely interest to the user, in accordance with stored rule data, and indicating said processor selection of the first depicted subject by a marking presented on the display;
capturing user speech with the microphone;
sending, to a speech recognition module, audio data corresponding to the user speech, and receiving recognized user speech data corresponding thereto;
the processor employing information from the recognized user speech data as a verbal clue to help identify a second physical subject depicted within the captured imagery, different than the processor-selected first depicted subject, that is of actual interest to said user;
the processor causing said marking to move from the depiction of the first subject to the depiction of the second subject;
after receiving the recognized user speech data, performing an image processing operation concerning the second depicted subject; and
presenting on said display, as a graphical overlay on the captured imagery, a graphical indicia output, different than said marking, that depends on a result of said image processing operation.

2. The method of claim 1 in which said verbal clue is a color.

3. The method of claim 1 in which said verbal clue is a person's name.

4. The method of claim 1 in which said verbal clue is a business' name.

5. The method of claim 1 in which said verbal clue comprises the word “left,” “right,” “up,” or “down”.

6. The method of claim 1 in which said recognized user speech data includes the word “square”.

7. A system comprising several elements including a processor, a memory, a camera, a display, and a microphone, at least certain of said elements being included in a face-worn apparatus, wherein the memory contains software instructions causing the system to perform the method of claim 1 .

8. A non-transitory computer readable medium containing software instructions operative to cause a user's portable computer system, equipped with a processor, a display, at least one microphone, and at least one image sensor, to perform acts including:
receiving camera imagery from the image sensor, depicting plural physical subjects within a physical environment of the user, and presenting the imagery to the user on the display;
selecting a first of said depicted plural physical subjects as being of likely interest to the user, in accordance with stored rule data, and indicating said selection of the first depicted subject by a marking presented on the display;
capturing user speech with the microphone;
sending, to a speech recognition module, audio data corresponding to the user speech, and receiving recognized user speech data corresponding thereto;
employing information from the recognized user speech data as a verbal clue to help identify a second physical subject depicted within the imagery, different than the first depicted subject, that is of actual interest to said user;
causing said marking to move from the depiction of the first subject to the depiction of the second subject;
after receiving the recognized user speech data, performing an image processing operation concerning the second depicted subject; and
presenting on said display, as a graphical overlay on the imagery, a graphical indicia output, different than said marking, that depends on a result of said image processing operation.

9. A method employing a portable user device having a display, at least one microphone that captures audio, and at least one image sensor for capturing imagery, the method comprising the acts:
(a) capturing imagery with the image sensor, the captured imagery depicting plural physical subjects within an environment of said user, and capturing user speech with the microphone;
(b) sending, to a speech recognition module, audio data corresponding to the user speech, and receiving recognized user speech data corresponding thereto;
(c) performing a verbally-clued computer-implemented cognition process, said cognition process employing information from the recognized user speech data as a verbal clue to help identify a physical subject within the captured imagery that is of interest to said user;
(d) displaying the captured imagery to the user on said display; and
(e) presenting on said display, as a graphical overlay on the captured imagery, a graphical indicia that is determined based on said verbally-clued cognition process;
wherein the presented graphical indicia varies based on said identified physical subject; and
wherein a set of stored rules establishes a priority order by which the cognition process ranks the plural subjects depicted in the captured imagery as being of probable interest to the user, wherein the recognized user speech data causes the cognition process to progress from one subject in said priority order, to a next subject in said priority order.

10. The method of claim 9 that further includes moving a bounding box from around said first subject, to said next subject, when the cognition process progresses from said first subject to said next subject.

11. The method of claim 9 in which said recognized user speech data includes the word “not”.

12. A method employing a portable user device having a display, at least one microphone that captures audio, and at least one image sensor for capturing imagery, the method comprising the acts:
(a) capturing imagery with the image sensor, the captured imagery depicting plural physical subjects within an environment of said user, and capturing user speech with the microphone;
(b) sending, to a speech recognition module, audio data corresponding to the user speech, and receiving recognized user speech data corresponding thereto;
(c) performing a verbally-clued computer-implemented cognition process, said cognition process employing information from the recognized user speech data as a verbal clue to help identify a physical subject within the captured imagery that is of interest to said user;
(d) displaying the captured imagery to the user on said display; and
(e) presenting on said display, as a graphical overlay on the captured imagery, a graphical indicia that is determined based on said verbally-clued cognition process;
wherein the presented graphical indicia varies based on said identified physical subject; and wherein:
the device indicates a first subject of possible interest to the user, by presenting an indicia on the screen indicating said first subject;
in response to first recognized user speech data, the device indicates a second subject of possible interest to the user, by changing said indicia to indicate said second subject instead of said first subject; and
in response to second recognized user speech data, the device indicates a third subject of possible interest to the user, by changing said indicia to indicate said third subject instead of said second subject;
wherein the method emulates conversation, with the user directing, the device responding, and the user further-directing.

Assignees

• Digimarc Corp

Inventors

• Davis Bruce L
• Rodriguez Tony F
• Rhoads Geoffrey B
• Conwell William Y
• Stach John

Classifications

• H04N23/66

  Remote control of cameras or camera parts, e.g. by remote control devices · CPC title

• H04N5/23203

  Electricity · mapped topic

• G06Q10/109

  Time management, e.g. calendars, reminders, meetings or time accounting · CPC title

• G10L15/22

  Procedures used during a speech recognition process, e.g. man-machine dialogue · CPC title

• G06K9/00671

  Physics · mapped topic