Determining gaze target based on facial features

Patent metadata

Field	Value
Publication number	US-10545574-B2
Application number	US-201715449058-A
Country	US
Kind code	B2
Filing date	Mar 3, 2017
Priority date	Mar 1, 2013
Publication date	Jan 28, 2020
Grant date	Jan 28, 2020

Abstract

A computer system can be controlled with non-contact inputs, such as eye-tracking devices. A visual indicator can be presented on a display to indicate the location where a computer function will take place (e.g., a common cursor). The visual indicator can be moved to a gaze target in response to continued detection of an action (e.g., touchpad touch) by a user for a predetermined period of time. A delay between the action and the movement of the visual indicator can allow a user time to “abort” movement of the visual indicator. Additionally, once the visual indicator has moved, the visual indicator can be controlled with additional precision as the user moves the gaze while continuing the action (e.g., continued holding of the touchpad).

First claim

What is claimed is:

1. A computing device, comprising: a display device for presenting a display; an image sensor configured to capture images of at least a portion of a user of the computing device; and at least one processor; wherein the at least one processor is configured to determine a first facial feature based on at least one captured image, the first facial feature comprising an eye lid position of an eye of the user, and the first facial feature being designated for control of the display in a vertical direction; determine a second facial feature based on the at least one captured image, the second facial feature being designated for control of the display in a horizontal direction, the second facial feature comprising a head pose; determine an orientation value, for the head pose of the user based on the at least one captured image, the orientation value indicating an offset of the head pose from a direct forward line of sight, wherein the orientation value is less than or equal to a threshold orientation value of at least seven degrees; in the event the orientation value does not exceed a threshold orientation value, determine a gaze target of the user on the display based on the first facial feature and the second facial feature by at least: determining a vertical component of the gaze target based on an openness of the eye lid position wherein: a first eye lid position at an upper location on the eye of the user corresponds with the gaze target being adjacent an upper portion of the display; and a second eye lid position at a lower location on the eye of the user corresponds with the gaze target being adjacent a bottom portion of the display; and determining a horizontal component of the gaze target based on the second facial feature; and scroll content displayed on the display based on the gaze target being within a predefined scrod zone on the display.

2. The computing device of claim 1 , wherein the at least one processor is further configured to determine the horizontal component based on the offset of the head pose.

3. The computing device of claim 1 , wherein the at least one processor is further configured to determine changes in the horizontal component based on the second facial feature.

4. The computing device of claim 1 , further comprising an infrared light source.

5. The computing device of claim 4 , wherein:
the image sensor is further configured to detect the infrared light emitted by the infrared light source, and
the at least one processor is further configured to determine the gaze target based on a reflection, from at least one eye, of the infrared light emitted by the infrared light source.

6. A method of providing user input to a computer device, the method comprising: determining a first facial feature based on at least one captured image, the first facial feature comprising an eye lid position of an eye of a user, and the first facial feature being designated for control of a display in a vertical direction; determining a second facial feature based on the at least one captured image, the second facial feature being designated for control of the display in a horizontal direction, the second facial feature comprising a head pose; determine an orientation value for the head pose of the user based on the at least one captured image, the orientation value indicating an offset of the head pose from a direct forward line of sight, wherein the orientation value is less than or equal to a threshold orientation value of at least seven degrees; in the event the orientation value does not exceed a threshold orientation value, determining a gaze target of the user on the display based on the first facial feature and the second facial feature by at least: determining a vertical component of the gaze target based solely on an openness of the eye lid position wherein: a first eye lid position at an upper location on the eye of the user corresponds with the gaze target being adjacent an upper portion of the display; and a second eye lid position at a lower location on the eye of the user corresponds with the gaze target being adjacent a bottom portion of the display; and determining a horizontal component of the gaze target based on the second facial feature; and scrolling content displayed on the display based on the gaze target being within a predefined scroll zone on the display.

7. The method of claim 6 , further comprising determining changes in the horizontal component based on the second facial feature.

8. The method of claim 6 , further comprising determining, based on the second facial feature, that the gaze target is within the predefined scroll zone on a display device.

9. A non-transitory computer-readable storage medium storing program instructions. wherein the program instructions, when executed by one or more processors, perform at least the steps of: determining a first facial feature based on at least one captured image, the first facial feature comprising an eye lid position of an eye of a user, and the first facial feature being designated for control of a display in a vertical direction; determining a second facial feature based on the at least one captured image, the second facial feature being designated for control of the display in a horizontal direction, the second facial feature comprising a head pose; determine an orientation value for the head pose of the user based on the at least one captured image, the orientation value indicating an offset of the head pose from a direct forward line of sight; in the event the orientation value is less than or equal to a threshold orientation value of at least seven degrees, determining a gaze target of the user on a display of a display device based on the first facial feature and the second facial feature by at least: determining a vertical component of the gaze target based solely on an openness of the eye lid position; and determining a horizontal component of the gaze target based on the second facial feature; and scrolling content displayed on the display based on the gaze target being within a predefined scroll zone on the display.

10. The computing device of claim 1 , wherein:
the at least one processor is further configured to determine that the user is reading elements on the display based on detecting saccades of the eye; and
scrolling the content displayed on the display is further based on determining that the user is reading the elements on the display.

11. The computing device of claim 1 , wherein the at least one processor is further configured to: determine a first facial feature based on at least one captured image, the first facial feature comprising an eye lid position of an eye of the user, and the first facial feature being designated for control of the display in a vertical direction; and in the event the orientation value does not exceed a threshold orientation value, determine a gaze target of the user on the display based on the first facial feature and the second facial feature by at least: determining a horizontal component of the gaze target based on the second facial feature; and determining a vertical component of the gaze target based on an openness of the eye lid position wherein; a first eye lid position at an upper location on the eye of the user corresponds with the gaze target being adjacent an upper portion of the display; and a second eye lid position at a lower location on the eye of the user corresponds with the gaze target being adjacent a bottom portion of the display; and scroll content displayed on the display based on the gaze target being within a predefined scroll zone on the display.

12. The computing device of claim 11 , wherein the at least one processor is further c

Assignees

• Tobii Ab

Inventors

• George-Svahn Erland
• Henderek David Figgins
• Lannsjö Rebecka
• Skogö Mårten
• Elvesjö John

Classifications

• G06F3/0481

  based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance · CPC title

• G06F3/02

  Input arrangements using manually operated switches, e.g. using keyboards or dials · CPC title

• G06F3/033

  Pointing devices displaced or positioned by the user {, e.g. mice, trackballs, pens or joysticks}; Accessories therefor (digitisers characterised by the transducing means G06F3/041) · CPC title

• G06F3/04845

  for image manipulation, e.g. dragging, rotation, expansion or change of colour · CPC title

• G06F3/0227

  Cooperation and interconnection of the input arrangement with other functional units of a computer (G06F3/023 - G06F3/037 take precedence) · CPC title