Method and device for eye tracking using event camera data

Patent metadata

Field	Value
Publication number	US-11474348-B2
Application number	US-202117481272-A
Country	US
Kind code	B2
Filing date	Sep 21, 2021
Priority date	Sep 28, 2017
Publication date	Oct 18, 2022
Grant date	Oct 18, 2022

Abstract

In one implementation, a method includes emitting light with modulating intensity from a plurality of light sources towards an eye of a user. The method includes receiving light intensity data indicative of an intensity of the plurality of glints reflected by the eye of the user in the form of a plurality of glints. The method includes determining an eye tracking characteristic of the user based on the light intensity data. In one implementation, a method includes generating, using an event camera comprising a plurality of light sensors at a plurality of respective locations, a plurality of event messages, each of the plurality of event messages being generated in response to a particular light sensor detecting a change in intensity of light and indicating a particular location of the particular light sensor. The method includes determining an eye tracking characteristic of a user based on the plurality of event messages.

First claim

What is claimed is:

1. An apparatus comprising:
an event camera comprising a plurality of light sensors at a plurality of respective locations, the event camera to generate, in response to particular light sensor detecting a change in intensity of light, an event message indicating a particular location of the particular light sensor;
a diplexer to filter the event messages from the event camera into a target-frequency event message and an off-target-frequency event message; and
an eye tracker to receive, from the diplexer, the target-frequency event message and the off-target-frequency event message, and determine an eye tracking characteristic of a user based on at least one of the target-frequency event message and the off-target-frequency event message.

2. The apparatus of claim 1 , wherein the event message further indicates a polarity of the change in intensity of light.

3. The apparatus of claim 1 , wherein the event message further indicates a time at which the change in intensity of light was detected.

4. The apparatus of claim 1 , wherein the particular location is indicated by a pixel coordinate.

5. The apparatus of claim 1 , wherein the eye tracker is to determine the eye tracking characteristic by detecting a pupil of an eye of the user.

6. The apparatus of claim 5 , wherein the eye tracker is to detect the pupil by:
generating an approximate intensity image of the eye of the user based on the off-target-frequency event message;
locating a low-intensity region in the approximate intensity image of the eye of user; and
fitting an ellipse to the low-intensity region.

7. The apparatus of claim 1 , wherein the eye tracker is to determine the eye tracking characteristic by tracking a pupil of an eye of the user.

8. The apparatus of claim 7 , wherein the eye tracker is to track the pupil of the eye of the user by:
locating one or more high-contrast edges based on the off-target-frequency event message; and
fitting a new ellipse based on the one or more high-contrast edges and a prior ellipse.

9. The apparatus of claim 1 , wherein the eye tracker is to determine the eye tracking characteristic by detecting one or more glints reflected from an eye of the user.

10. The apparatus of claim 9 , wherein the eye tracker is to detect the one or more glints based on event messages that indicate an increase in intensity of light without being based on event message that indicate a decrease in intensity of light.

11. The apparatus of claim 9 , wherein the eye tracker is to detect the one or more glints based on event messages that indicate a decrease in intensity of light without being based on event message that indicate an increase in intensity of light.

12. The apparatus of claim 1 , further comprising a plurality of light sources to emit light with modulating intensity towards an eye of the user, wherein the event camera is to generate light intensity data indicative of an intensity of the emitted light reflected by the eye of the user in the form of a plurality of glints, and the eye tracker is to determine the eye tracking characteristic of the user based on the light intensity data.

13. A method comprising:
generating, using an event camera comprising a plurality of light sensors at a plurality of respective locations, a plurality of event messages, each of the plurality of event messages being generated in response to a particular light sensor detecting a change in intensity of light and indicating a particular location of the particular light sensor;
filtering the plurality of event messages into a target-frequency event message and an off-target-frequency event message; and
determining an eye tracking characteristic of a user based on at least one of the target-frequency event message and the off-target-frequency event message.

14. The method of claim 13 , wherein determining the eye tracking characteristic includes detecting a pupil of an eye of the user.

15. The method of claim 14 , wherein detecting the pupil includes:
generating an approximate intensity image of the eye of the user based on the off-target-frequency event message;
locating a low-intensity region in the approximate intensity image of the eye of user; and
fitting an ellipse to the low-intensity region.

16. The method of claim 13 , wherein determining the eye tracking characteristic includes tracking a pupil of an eye of the user.

17. The method of claim 16 , wherein tracking the pupil of the eye of the user includes:
locating one or more high-contrast edges based on the off-target-frequency event message; and
fitting a new ellipse based on the one or more high-contrast edges and a prior ellipse.

18. The method of claim 13 , further comprising emitting light with modulating intensity from a plurality of light sources towards an eye of the user, wherein the event messages are indicative of light intensity data indicative of an intensity of the emitted light reflected by the eye of the user in the form of a plurality of glints, and wherein determining the eye tracking characteristic of the user is based on the light intensity data.

19. A non-transitory computer-readable medium having instructions encoded thereon which, when executed by a processor, cause the processor to perform operations comprising:
generating, using an event camera comprising a plurality of light sensors at a plurality of respective locations, a plurality of event messages, each of the plurality of event messages being generated in response to a particular light sensor detecting a change in intensity of light and indicating a particular location of the particular light sensor;
filtering the plurality of event messages into a target-frequency event message and an off-target-frequency event message; and
determining an eye tracking characteristic of a user based on at least one of the target-frequency event message and the off-target-frequency event message.

20. The non-transitory computer-readable medium of claim 19 , wherein the operations further comprise emitting, via a plurality of light sources, light with modulating intensity towards an eye of the user, wherein the event messages are indicative of light intensity data indicative of an intensity of the emitted light reflected by the eye of the user in the form of a plurality of glints, and wherein determining the eye tracking characteristic of the user is based on the light intensity data.

Assignees

• Apple Inc

Inventors

• Petljanski Branko
• Bedikian Raffi A
• Kurz Daniel
• Gebauer Thomas
• Jia Li

Classifications

• G02B27/0093 Primary

  with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking · CPC title

• G02B27/0179

  Display position adjusting means not related to the information to be displayed · CPC title

• G06F3/147

  using display panels · CPC title

• G02B2027/0187

  slaved to motion of at least a part of the body of the user, e.g. head, eye · CPC title

• G09G3/003

  to produce spatial visual effects · CPC title