Visual display with illuminators for gaze tracking

Patent metadata

Field	Value
Publication number	US-10191542-B2
Application number	US-201715483298-A
Country	US
Kind code	B2
Filing date	Apr 10, 2017
Priority date	Apr 1, 2009
Publication date	Jan 29, 2019
Grant date	Jan 29, 2019

Abstract

A visual display includes hidden reference illuminators adapted to emit invisible light for generating corneo-scleral reflections on an eye watching a screen surface of the display. The tracking of such reflections and the pupil center provides input to gaze tracking. A method for equipping and an LCD with a reference illuminator are also provided. Also provides are a system and method for determining a gaze point of an eye watching a visual display that includes reference illuminators. The determination of the gaze point may be based on an ellipsoidal cornea model.

First claim

The invention claimed is:

1. A method of determining a gaze direction of an eye watching a visual display, the method comprising:
selecting either a bright-pupil imaging mode or a dark-pupil imaging mode;
determining an image sensor to use for gaze direction determination;
selectively illuminating an eye of a user using a plurality of reference illuminators embedded beneath a screen of a display device;
determining a location of a reflection on the eye from at least one of the plurality of reference illuminators;
determining a particular reference illuminator of the plurality of reference illuminators to use for gaze direction determination based on:
whether the bright-pupil imaging mode or the dark-pupil imaging mode is selected;
the image sensor selected; and
the location of the reflection on the eye from the particular reference illuminator being nearer to a pupil center of the eye than a remainder of the plurality of reference illuminators; and

determining a gaze direction of the eye based on the image sensor selected and the reflection from the particular reference illuminator.

2. The method of claim 1 , wherein:
the plurality of reference illuminators are adapted to emit light having a wavelength of at least 700 nm.

3. The method of claim 1 , wherein:
the plurality of reference illuminators are adapted to emit infrared or near-infrared light.

4. The method of claim 1 , wherein the step of determining a particular reference illuminator to use for gaze direction determination is further based on:
the particular reference illuminator generating a reflection located on a cornea of the eye.

5. The method of claim 1 , wherein the method further comprises:
determining a deviation from the pupil center of the eye to the reflection on the eye from the particular reference illuminator.

6. The method of claim 5 , wherein the step of determining a particular reference illuminator to use for gaze direction determination is further based on:
the deviation from the pupil center of the eye to the reflection on the eye from the particular reference illuminator.

7. The method of claim 1 , further comprising the steps of:
selecting an initial reference illuminator of the plurality of reference illuminators;
determining the location of a reflection from the initial reference illuminator relative to the eye;
determining a deviation from the pupil center of the eye to the reflection from the initial reference illuminator;
determining whether there is an alternative reference illuminator, which has such a position to the initially selected reference illuminator that a more centric reflection from the alternative reference illuminator on the pupil of the eye can be achieved; and
if such an alternative reference illuminator is available, then determining to use the alternative reference illuminator for gaze point determination.

8. The method of claim 1 , further comprising:
acquiring an image of the eye including a corneo-scleral reflection of at least one of the reference illuminators, and wherein determining the location of the reflection on the eye of at least one of the plurality of reference illuminators is made based on the image.

9. The method of claim 1 , wherein determining the image sensor to use for gaze direction determination comprises:
acquiring a plurality of images using the image sensor;
evaluating an image quality metric of the plurality of images; and
selecting the image sensor when the image quality metric of the image sensor is preferable over an image quality metric of an image acquired by a separate image sensor.

10. The method of claim 9 , wherein the image quality metric is based on a factor selected from the group consisting of:
a number of pupils detected;
a gaze detection noise value;
a pupil contrast;
an iris gradient;
an obstacle; and
a signal to noise ratio.

11. The method of claim 1 , wherein selecting either the bright-pupil imaging mode or the dark-pupil imaging mode comprises:
acquiring an image of the eye in each of the bright-pupil imaging mode and the dark-pupil imaging mode;
comparing an image quality metric of the image in the bright-pupil imaging mode to an image quality metric of the image in the dark-pupil imaging mode;
selecting the bright-pupil imaging mode when the image quality metric of the image in the bright-pupil imaging mode is preferable to the image quality metric of the image in the dark-pupil imaging mode; and
selecting the dark-pupil imaging mode when the image quality metric of the image in the dark-pupil imaging mode is preferable to the image quality metric of the image in the bright-pupil imaging mode.

12. The method of claim 1 , wherein determining the reference illuminator to use for gaze direction determination being based on whether the bright-pupil imaging mode or the dark-pupil imaging mode is selected comprises:
selecting the reference illuminator which is coaxial with the image sensor when the bright-pupil imaging mode is selected; and
selecting the reference illuminator which is non-coaxial with the image sensor when the dark-pupil imaging mode is selected.

13. A non-transitory computer-readable medium having instructions stored thereon executable by a computing device to cause the computing device to perform operations comprising:
selecting either a bright-pupil imaging mode or a dark-pupil imaging mode;
determining an image sensor to use for gaze direction determination;
selectively illuminating an eye of a user using a plurality of reference illuminators embedded beneath a screen of a display device;
determining a location of a reflection on the eye from at least one of the plurality of reference illuminators;
determining a particular reference illuminator of the plurality of reference illuminators to use for gaze direction determination based on:
whether the bright-pupil imaging mode or the dark-pupil imaging mode is selected;
the image sensor selected; and
the location of the reflection on the eye from the particular reference illuminator being nearer to a pupil center of the eye than a remainder of the plurality of reference illuminators; and

determining a gaze direction of the eye based on the image sensor selected and the reflection from the particular reference illuminator.

14. The non-transitory computer-readable medium of claim 13 , wherein:
the plurality of reference illuminators are adapted to emit light having a wavelength of at least 700 nm.

15. The non-transitory computer-readable medium of claim 13 , wherein:
the plurality of reference illuminators are adapted to emit infrared or near-infrared light.

16. A system for determining a gaze point of an eye watching a visual display, the system comprising:
a plurality of image sensors;
a display device;
a plurality of reference illuminators embedded beneath a screen of the display device;
one or more processors configured to
select either a bright-pupil imaging mode or a dark-pupil imaging mode;
determine an image sensor to use for gaze direction determination from the plurality of image sensors;
selectively illuminate an eye of a user using the plurality of reference illuminators;
determine a location of a reflection on the eye from at least one of the plurality of reference illuminators;
determine a particular reference illuminator of the plurality of reference illuminators to use for gaze direction determination based on:
whether the bright-pupil imaging mode or the dark-pupil imaging mode is selected;
the image sensor selected; and
the location of the reflection on the eye from the par

Assignees

• Tobii Ab

Inventors

• Blixt Peter
• Kingback Anders
• Rehnstrom Bengt
• Elvesjö John
• Kuldkepp Mattias

Classifications

• G06F3/01

  Input arrangements or combined input and output arrangements for interaction between user and computer (G06F3/16 takes precedence) · CPC title

• A61B3/113

  for determining or recording eye movement · CPC title

• G06K9/00604

  Physics · mapped topic

• G06F3/013 Primary

  Eye tracking input arrangements (G06F3/015 takes precedence) · CPC title

• G01S17/46

  Indirect determination of position data · CPC title