Affecting the behavior of a user device based on a user's gaze

Patent metadata

Field	Value
Publication number	US-9526127-B1
Application number	US-201213676517-A
Country	US
Kind code	B1
Filing date	Nov 14, 2012
Priority date	Nov 18, 2011
Publication date	Dec 20, 2016
Grant date	Dec 20, 2016

Abstract

A device may determine whether a user is facing a display screen associated with the device; and present feedback to the user. When presenting the feedback, the device may present visual information, that is based on the feedback, when determining that the user is facing the display screen associated with the device, and present audio information, that is based on the feedback, when determining that the user is not facing the display screen associated with the device. At least a portion of the audio information might not be presented when the visual information is presented when the user is facing the display screen associated with the device.

First claim

What is claimed is:

1. A computer-implemented method comprising:
receiving, by a user device, audio data corresponding to an utterance of a user;
determining that a first portion of a transcription of the utterance includes a keyword that is associated with a voice command;
after determining that the first portion of the transcription of the utterance includes the keyword that is associated with the voice command, determining, by the user device, that an image of the user does not include one or more features that are characteristic of the user facing a display of the user device; and
in response to determining that the image of the user does not include one or more features that are characteristic of the user facing the display of the user device, determining, by the user device, to prevent a second portion of the transcription of the utterance from being processed as a voice command.

2. The method of claim 1 , comprising:
discarding the second portion of the transcription without inputting the second portion of the transcription to a dialog engine based on determining to prevent the second portion of the transcription of the utterance from being processed as a voice command.

3. The method of claim 1 , comprising:
generating the image of the user by a camera on the user device after determining that the first portion of the transcription of the utterance includes the keyword that is associated with the voice command.

4. The method of claim 1 , wherein the voice command is a command for the user to take an action other than taking a picture.

5. The method of claim 1 , wherein determining the image does not include one or more features that are characteristic of the user facing a display of the user device comprises:
determining a direction of a gaze of the user; and
classifying the gaze of the user as a gaze that is not directed toward the display of the user device.

6. The method of claim 1 , wherein determining that the image does not include one or more features that are characteristic of the user facing a display of the user device comprises:
determining that one or more particular facial features are not visible in the image.

7. The method of claim 1 , wherein determining that the image does not include one or more features that are characteristic of the user facing a display of the user device comprises:
determining an orientation of two or more particular facial features of the user with respect to each other.

8. The method of claim 1 , wherein determining that the image includes one or more features that are characteristic of the user facing a display of the user device comprises:
comparing the image to a different image in which the user is labeled as facing the display of the mobile device.

9. A system comprising:
one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:
receiving, by a user device, audio data corresponding to an utterance of a user;
determining that a first portion of a transcription of the utterance includes a keyword that is associated with a voice command;
after determining that the first portion of the transcription of the utterance includes the keyword that is associated with the voice command, determining, by the user device, that an image of the user does not include one or more features that are characteristic of the user facing a display of the user device; and
in response to determining that the image of the user does not include one or more features that are characteristic of the user facing the display of the user device, determining, by the user device, to prevent a second portion of the transcription of the utterance from being processed as a voice command.

10. The system of claim 9 , wherein the operations further comprise:
discarding the second portion of the transcription without inputting the second portion of the transcription to a dialog engine based on determining to prevent the second portion of the transcription of the utterance from being processed as a voice command.

11. The system of claim 9 , wherein the operations further comprise:
generating the image of the user by a camera on the user device after determining that the first portion of the transcription of the utterance includes the keyword that is associated with the voice command.

12. The system of claim 9 , wherein the voice command is a command for the user to take an action other than taking a picture.

13. The system of claim 9 , wherein determining the image does not include one or more features that are characteristic of the user facing a display of the user device comprises:
determining a direction of a gaze of the user; and
classifying the gaze of the user as a gaze that is not directed toward the display of the user device.

14. The system of claim 9 , wherein determining that the image does not include one or more features that are characteristic of the user facing a display of the user device comprises:
determining that one or more particular facial features are not visible in the image.

15. The system of claim 9 , wherein determining that the image does not include one or more features that are characteristic of the user facing a display of the user device comprises:
determining an orientation of two or more particular facial features of the user with respect to each other.

16. The system of claim 9 , wherein determining that the image includes one or more features that are characteristic of the user facing a display of the user device comprises:
comparing the image to a different image in which the user is labeled as facing the display of the mobile device.

17. A non-transitory computer-readable medium storing software comprising instructions executable by one or more computers which, upon such execution, cause the one or more computers to perform operations comprising:
receiving, by a user device, audio data corresponding to an utterance of a user;
determining that a first portion of a transcription of the utterance includes a keyword that is associated with a voice command;
after determining that the first portion of the transcription of the utterance includes the keyword that is associated with the voice command, determining, by the user device, that an image of the user does not include one or more features that are characteristic of the user facing a display of the user device; and
in response to determining that the image of the user does not include one or more features that are characteristic of the user facing the display of the user device, determining, by the user device, to prevent a second portion of the transcription of the utterance from being processed as a voice command.

18. The medium of claim 17 , wherein the operations further comprise:
discarding the second portion of the transcription without inputting the second portion of the transcription to a dialog engine based on determining to prevent the second portion of the transcription of the utterance from being processed as a voice command.

19. The medium of claim 17 , wherein the operations further comprise:
generating the image of the user by a camera on the user device after determining that the first portion of the transcription of the utterance includes the keyword that is associated with the voice command.

20. The medium of claim 17 , wherein the voice command is a command for the user to take an action other than taking a picture

Assignees

• Google Inc

Inventors

• Taubman Gabriel
• Byrne William J

Classifications

• G06F3/167 Primary

  Audio in a user interface, e.g. using voice commands for navigating, audio feedback · CPC title

• A63F2300/1093

  using visible light · CPC title

• G10L15/25

  using position of the lips, movement of the lips or face analysis · CPC title

• H04W88/02 Primary

  Terminal devices · CPC title

• G10L25/78

  Detection of presence or absence of voice signals (switching of direction of transmission by voice frequency in two-way loud-speaking telephone systems H04M9/10) · CPC title