System for gaze interaction

Patent metadata

Field	Value
Publication number	US-10324528-B2
Application number	US-201815985049-A
Country	US
Kind code	B2
Filing date	May 21, 2018
Priority date	Jan 4, 2012
Publication date	Jun 18, 2019
Grant date	Jun 18, 2019

Abstract

A method and system for assisting a user when interacting with a graphical user interface combines gaze based input with gesture based user commands. A user of a computer system without a traditional touch-screen can interact with graphical user interfaces in a touch-screen like manner using a combination of gaze based input and gesture based user commands. A solution for touch-screen like interaction uses gaze input and gesture based input as a complement or an alternative to touch-screen interactions with a computer device having a touch-screen. Combined gaze and gesture based interaction with graphical user interfaces can be used to achieve a touch-screen like environment in computer systems without a traditional touchscreen or in computer systems having a touchscreen arranged ergonomically unfavorable for the user or a touchscreen arranged such that it is more comfortable for the user to use gesture and gaze for the interaction than the touchscreen.

First claim

The invention claimed is:

1. A system for generating gesture based commands during user interaction with an information presentation area, the system comprising:
a processor; and
a memory storing computer-readable instructions that, upon execution by the processor, configure the system to:
receive gaze data from a gaze tracking module, the gaze data indicating a gaze point of a user on the information presentation area;
receive, from an input device, first user gesture data;
determine, based on the first user gesture data and the gaze data, a gaze point area on the information presentation area, the gaze point area including the gaze point;
receive, from the input device, second user gesture data, the second user gesture data received subsequently to receiving the first user gesture data and while the user gaze data are received; and
execute at least one user action manipulating the gaze point area based on the second user gesture data, the user action comprising resizing the gaze point area to generate an updated gaze point area, the updated gaze point area including the gaze point and overlapping with the gaze point area.

2. The system of claim 1 , wherein the input device comprises a physical touch sensitive area, wherein the first user gesture data corresponds to a press of a finger on the physical touch sensitive area, and wherein the second user gesture data corresponds to a movement of the finger on the physical touch sensitive area.

3. The system of claim 2 , wherein the execution of the computer-readable instructions further configure the system to:
receive, from the input device, third user gesture data indicating a release of the finger from the physical touch sensitive area, the movement of the finger followed next by the release of the finger;
determine, based on an object being included in the updated gaze point area, that the third user gesture data is associated with a select command to select the object; and
select, based on the select command, the object in the updated gaze point area.

4. The system of claim 3 , wherein the execution of the computer-readable instructions further configure the system to:
receive, from the input device, fourth user gesture data indicating a first finger press on the physical touch sensitive area; and
present, at the gaze point, an icon illustrating a location of the gaze point on the information presentation area.

5. The system of claim 4 , wherein the execution of the computer-readable instructions further configure the system to:
receive, from the input device, fifth user gesture data indicating a finger movement on the physical touch sensitive area, the first finger press followed next by the finger movement;
determine, based on the icon, that the fifth gesture data is associated with a relocation command to relocate the gaze point to a new location on the information presentation area; and
relocate, based on the relocation command, the gaze point from the location to the new location.

6. The system of claim 5 , wherein the execution of the computer-readable instructions further configure the system to:
receive, based on the relocating, sixth user gesture data indicating a second finger press concurrent with the first finger press, the finger movement followed next by the second finger press; and
manipulate the object based on the sixth user gesture data and the new location of the gaze point.

7. The system of claim 2 , wherein the execution of the computer-readable instructions further configure the system to:
receive, from the input device, third user gesture data indicating a finger tap on the physical touch sensitive area, the movement of the finger followed next by the finger tap;
determine, based on an object being included in the updated gaze point area, that the third user gesture data is associated with a select command to select the object; and
select, based on the select command, the object in the updated gaze point area.

8. The system of claim 1 , wherein the input device comprises a physical touch sensitive area, wherein the first user gesture data corresponds to a press of a finger on the physical touch sensitive area, wherein the second user gesture data corresponds to a circular movement of the finger on the physical touch sensitive area, and wherein the user action comprises a zoom in or a zoom out depending on a direction of the circular movement.

9. The system of claim 1 , wherein the input device comprises a physical touch sensitive area, wherein the first user gesture data corresponds to a press of a finger on the physical touch sensitive area, wherein the second user gesture data corresponds to holding the press of the finger and a movement of a different finger on the physical touch sensitive area, and wherein the user action comprises a zoom in or a zoom out depending on a direction of the movement.

10. The system of claim 1 , wherein the input device comprises a physical touch sensitive area, wherein the first user gesture data corresponds to a press of two fingers on the physical touch sensitive area, wherein the second user gesture data corresponds to movement of the two fingers in opposite direction on the physical touch sensitive area, and wherein the user action comprises a zoom in or a zoom out.

11. The system of claim 1 , wherein the input device comprises a physical touch sensitive area, wherein the first user gesture data corresponds to a finger tap on the physical touch sensitive area, wherein the second user gesture data corresponds to holding the press of the finger and a movement of a different finger on the physical touch sensitive area, and wherein the user action comprises sliding the information presentation area.

12. The system of claim 1 , wherein the input device comprises a physical touch sensitive area, wherein the first user gesture data corresponds to a finger tap on the physical touch sensitive area, wherein the second user gesture data corresponds to a different touch on the physical touch sensitive area, and wherein the user action comprises panning the information presentation area based on the gaze point.

13. The system of claim 1 , wherein the input device comprises a physical touch sensitive area, wherein the gaze point corresponds to an object in the presentation area, wherein the first user gesture data corresponds to a press of two fingers on the physical touch sensitive area, wherein the second user gesture data corresponds to movement of the two fingers in opposite direction on the physical touch sensitive area, and wherein the user action comprises rotating the object.

14. The system of claim 1 , wherein the execution of the computer-readable instructions further configure the system to present, based on the first user gesture data and prior to receiving the second user gesture data, a virtual finger on the information presentation area, wherein the second user gesture data corresponds to a selection and movement of the virtual finger on the information presentation area.

15. The system of claim 14 , wherein the execution of the computer-readable instructions further configure the system to receive, based on the selection and movement of the virtual finger, a third user gesture data indicating a point of action corresponding to an updated location of the virtual finger on the information presentation area, wherein the user action is executed based on the point of action.

16. A method for generating gesture based commands during user interaction with an information presentation area, the method implemented by a computer system and comprising:
receiving gaze data from

Assignees

• Tobii Ab

Inventors

• Cederlund Markus
• Gavelin Robert
• Vennström Anders
• Kaplan Anders
• Olsson Anders
• Skogö Mårten

Classifications

• G02B2027/0187

  slaved to motion of at least a part of the body of the user, e.g. head, eye · CPC title

• G02B27/01

  Head-up displays · CPC title

• G06F3/017 Primary

  Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883) · CPC title

• G06F2203/0381

  Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer · CPC title

• G02B27/017

  Head mounted · CPC title