Method and device for eye tracking using event camera data

Patent metadata

Field	Value
Publication number	US-12105280-B2
Application number	US-202217961963-A
Country	US
Kind code	B2
Filing date	Oct 7, 2022
Priority date	Sep 28, 2017
Publication date	Oct 1, 2024
Grant date	Oct 1, 2024

Abstract

In one implementation, a method includes emitting light with modulating intensity from a plurality of light sources towards an eye of a user. The method includes receiving light intensity data indicative of an intensity of the plurality of glints reflected by the eye of the user in the form of a plurality of glints. The method includes determining an eye tracking characteristic of the user based on the light intensity data. In one implementation, a method includes generating, using an event camera comprising a plurality of light sensors at a plurality of respective locations, a plurality of event messages, each of the plurality of event messages being generated in response to a particular light sensor detecting a change in intensity of light and indicating a particular location of the particular light sensor. The method includes determining an eye tracking characteristic of a user based on the plurality of event messages.

First claim

What is claimed is:

1. A method comprising:
emitting light according to a modulation characteristic from one or more light sources towards an eye;
receiving, from an event camera, an event message in response to detecting a change in an intensity of the emitted light reflected by the eye; and
determining an eye tracking characteristic based on the event message and the modulation characteristic.

2. The method of claim 1 , wherein the event message includes information regarding a corresponding one of the one or more of light sources.

3. The method of claim 2 , wherein the event message indicates a respective location of the corresponding one of the one or more light sources.

4. The method of claim 2 , wherein the event message indicates a change in intensity of the corresponding one of the one or more light sources.

5. The method of claim 4 , wherein the event message further indicates a polarity of the change in the intensity.

6. The method of claim 4 , wherein the event message further indicates a time at which the change in the intensity of light was detected.

7. The method of claim 1 , wherein the event camera is different from a frame camera that generates one or more video frames, and wherein the event message does not include information regarding the one or more video frames.

8. The method of claim 1 , wherein the modulation characteristic corresponds to a modulation intensity of the emitted light.

9. The method of claim 1 , wherein the modulation characteristic corresponds to a modulation frequency of the emitted light.

10. The method of claim 1 , wherein the modulation characteristic corresponds to a modulation phase of the emitted light.

11. The method of claim 1 , wherein the event message includes light intensity data and wherein determining the eye tracking characteristic includes filtering the light intensity data according to the modulation characteristic.

12. The method of claim 11 , wherein filtering the light intensity data includes filtering out light intensity data outside a frequency range of the modulation characteristic to generate target-frequency light intensity data, and filtering out light intensity data inside the frequency range of the modulation characteristic to generate off-target-frequency light intensity data.

13. The method of claim 1 , wherein determining the eye tracking characteristic includes detecting a pupil of the eye.

14. The method of claim 13 , wherein detecting the pupil of the eye includes:
generating an approximate intensity image of the eye based on the event message;
locating a low-intensity region in the approximate intensity image of the eye; and
fitting an ellipse to the low-intensity region.

15. The method of claim 14 , wherein detecting the pupil of the eye includes:
locating one or more high-contrast edges based on the event message; and
fitting a new ellipse based on the one or more high-contrast edges and a prior ellipse.

16. The method of claim 1 , wherein the eye tracking characteristic includes a gaze direction and/or a blinking state.

17. A system comprising:
one or more light sources to emit light according to a modulation characteristic towards an eye;
an event camera to generate an event message in response to detecting a change in an intensity of the emitted light reflected by the eye; and
a processer to determine an eye tracking characteristic based on the event message and the modulation characteristic.

18. The system of claim 17 , wherein the event message includes information regarding a corresponding one of the one or more of light sources.

19. The system of claim 17 , wherein the event camera is different from a frame camera that generates one or more video frames, and wherein the event message does not include information regarding the one or more video frames.

20. The system of claim 17 , wherein the event message includes information regarding a corresponding one of the one or more of light sources.

21. A non-transitory computer-readable medium having instructions encoded thereon which, when executed by a processor, cause the processor to perform operations comprising:
emitting light according to a modulation characteristic from one or more light sources towards an eye;
receiving, from an event camera, an event message in response to detecting a change in an intensity of the emitted light reflected by the eye; and
determining an eye tracking characteristic based on the event message and the modulation characteristic.

22. The non-transitory computer-readable medium of claim 21 , wherein the event message includes information regarding a corresponding one of the one or more of light sources.

23. The non-transitory computer-readable medium of claim 21 , wherein the event camera is different from a frame camera that generates one or more video frames, and wherein the event message does not include information regarding the one or more video frames.

24. The non-transitory computer-readable medium of claim 21 , wherein the modulation characteristic corresponds to a modulation intensity, a modulation frequency or a modulation phase of the emitted light.

Assignees

• Apple Inc

Inventors

• Petljanski Branko
• Bedikian Raffi A
• Kurz Daniel
• Gebauer Thomas
• Jia Li

Classifications

• G06F3/013 Primary

  Eye tracking input arrangements (G06F3/015 takes precedence) · CPC title

• G02B2027/0187

  slaved to motion of at least a part of the body of the user, e.g. head, eye · CPC title

• G02B2027/014

  comprising information/image processing systems · CPC title

• G02B2027/0138

  comprising image capture systems, e.g. camera · CPC title

• G02B27/0179

  Display position adjusting means not related to the information to be displayed · CPC title