Interface with gaze detection and voice input

Patent metadata

Field	Value
Publication number	US-9250703-B2
Application number	US-201113110817-A
Country	US
Kind code	B2
Filing date	May 18, 2011
Priority date	Mar 6, 2006
Publication date	Feb 2, 2016
Grant date	Feb 2, 2016

Abstract

Methods, computer programs, and systems for interfacing a user with a computer program, utilizing gaze detection and voice recognition, are provided. One method includes an operation for determining if a gaze of a user is directed towards a target associated with the computer program. The computer program is set to operate in a first state when the gaze is determined to be on the target, and set to operate in a second state when the gaze is determined to be away from the target. When operating in the first state, the computer program processes voice commands from the user, and, when operating in the second state, the computer program omits processing of voice commands.

First claim

What is claimed is:

1. A method for interfacing with a computer program executing on a computing device, the method comprising:
presenting a character on a display;
determining if a point of gaze (POG) of a user is on the display;
setting the character to present a first behavior when the POG is determined to be on the display, wherein the first behavior provides an impression of awareness of the user by the character, wherein a gaze of the character is maintained toward the user and tracked while the POG of the user is on the character, wherein the tracking causes the character to show motion that enables the maintaining of the gaze of the character toward the user;
setting the character to present a second behavior when the POG is determined to be away from the display, wherein the second behavior provides an impression that the character is not interacting with the user, wherein the gaze of the character is not tracked while the POG of the user is away from the character, wherein the computer program processes voice commands from the user when operating in the first behavior, and wherein the computer program omits processing of voice commands when operating in the second behavior; and
setting the character to operate in a third behavior when multiple users are detected, wherein the character, when operating in the third behavior, performs a validation operation after receiving a voice command, the validation operation including asking a question to the user to validate the voice command, wherein the validation operation is not performed when the voice command is received while the character is presenting the first behavior.

2. The method as recited in claim 1 , wherein the computer program is a game executing on a game console, and wherein the voice command is an input for the game to perform an operation related to the character.

3. The method as recited in claim 1 , wherein determining if the POG of the user is directed towards the display further includes:
performing image analysis for a plurality of images of a video stream; and
detecting facial features of the user based on the image analysis.

4. The method as recited in claim 1 , wherein determining if the POG of the user is directed towards the display further includes:
analyzing corneal reflection to determine a direction of the gaze of the user.

5. The method as recited in claim 1 , wherein the method operations are performed by the computer program when executed by one or more processors, the computer program being embedded in a non-transitory computer-readable storage medium.

6. The method as recited in claim 1 , further including:
identifying a plurality of gaze targets on the display, each gaze target corresponding to a menu selection;
detecting the POG on a first gaze target from the plurality of gaze targets;
highlighting on the display a first menu selection corresponding to the first gaze target to differentiate the first menu selection with respect to other menu selections, wherein the user utilizes gaze to select from the menu selections; and
processing voice commands to initiate an action for the first menu selection.

7. The method as recited in claim 1 , wherein operating in the first behavior further includes:
processing controller inputs to perform corresponding actions on the character.

8. The method as recited in claim 1 , wherein the first behavior further includes one or more of the character providing eye contact, or showing excitement, or smiling, or jumping, or providing verbal cues to the user.

9. The method as recited in claim 1 , wherein the second behavior further includes the character acting in a passive way showing lack of enthusiasm and awareness of the user.

10. The method as recited in claim 1 , further including:
tracking a location of a controller using captured images with a camera;
determining which user from a plurality of users in front of the display is holding the controller based on the captured images; and
determining the gaze of the user holding the controller.

11. A method for controlling interfacing reactions with game objects produced by a computer program and rendered on a display, comprising:
detecting presence of a user in a zone defined in front of an image capture device;
presenting a character on a display;
determining if a point of gaze (POG) of the user is on the character on the display;
setting the character to present a first behavior when the POG is on the character on the display, wherein the first behavior provides an impression of awareness of the user by the character, wherein a gaze of the character is maintained toward the user and tracked while the POG of the user is on the character, wherein the tracking causes the character to show motion that enables the maintaining of the gaze of the character toward the user, the first behavior including processing of user voice commands by the computer program to perform actions on the character;
setting the character to present a second behavior when the POG is away from the character, wherein the second behavior provides an impression that the character is not interacting with the user, wherein the gaze of the character is not tracked while the POG of the user is away from the character, the second behavior including an omission of processing of user voice commands by the computer program; and
setting the character to operate in a third behavior when multiple users are detected, wherein the character, when operating in the third behavior, performs a validation operation after receiving a voice command, the validation operation including asking a question to the user to validate the voice command, wherein the validation operation is not performed when the voice command is received while the character is presenting the first behavior.

12. The method as recited in claim 11 , wherein determining if the POG is on the character further includes:
analyzing reflection from pupils of the user from an infrared light.

13. The method as recited in claim 11 , wherein the first behavior further includes taking turns speaking to the user.

14. The method as recited in claim 11 , further including:
tracking a location of a controller using captured images with a camera;
determining which user from a plurality of users in front of the display is holding the controller based on the captured images; and
determining the gaze of the user holding the controller.

15. The method as recited in claim 11 , further including:
identifying a plurality of gaze targets on the display, each gaze target corresponding to a menu selection;
detecting the POG on a first gaze target from the plurality of gaze targets;
highlighting on the display a first menu selection corresponding to the first gaze target, wherein the user utilizes gaze to select from the menu selections; and
processing voice commands to initiate an action for the first menu selection.

16. An apparatus with a visual interface for capturing user input to a computer program executing on a computing device, the apparatus comprising:
a camera that captures images of a user situated in front of a display, the computer program presenting a character on the display;
a gaze detection module that determines if a point of gaze (POG) of a user is directed towards the character on the display based on the captured images, wherein the computer program sets the character to present a first behavior when the POG is on the character on the display, and wherein the computer program sets the character to present a second behavior when the gaze is determined to be away from the ch

Assignees

• Hernandez-Abrego Gustavo A
• Osman Steven
• Mikhailov Anton
• Chen Ruxin
• Sony Computer Entertainment Inc

Inventors

• Hernandez-Abrego Gustavo A
• Osman Steven
• Mikhailov Anton
• Chen Ruxin

Classifications

• A63F13/10

  Human Necessities · mapped topic

• G06F3/167 Primary

  Audio in a user interface, e.g. using voice commands for navigating, audio feedback · CPC title

• G06F2203/0381

  Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer · CPC title

• B60K37/06

  Operations & Transport · mapped topic

• A63F2300/308

  Details of the user interface · CPC title