Systems and methods for eye tracking in virtual reality and augmented reality applications

Patent metadata

Field	Value
Publication number	US-11622103-B2
Application number	US-201916970549-A
Country	US
Kind code	B2
Filing date	Feb 15, 2019
Priority date	Feb 15, 2018
Publication date	Apr 4, 2023
Grant date	Apr 4, 2023

Abstract

An augmented reality, virtual reality, or other wearable apparatus comprises an eye tracking device comprising an image sensor, a lens, and one or more processors. In some embodiments, the lens comprises a marker, and the one or more processors are configured to receive an image from the image sensor, wherein the image shows the marker, determine a distance from the image sensor to the marker based on the image, and change a calibration parameter of an eye tracking algorithm based on the distance. In some embodiments, the one or more processors are configured to receive image data from the image sensor, wherein the image data corresponds to an image as observed through the lens, determine a level or pattern of pincushion distortion in the image based on the image data, and change a calibration parameter of an eye tracking algorithm based on the level or the pattern of pincushion distortion.

First claim

What is claimed is:

1. A wearable augmented reality or virtual reality headset, wherein the headset comprises:
an eye tracking device comprising:
an image sensor, and
a first lens comprising a first marker; and

one or more processors configured to at least:
receive a first image from the image sensor, wherein the first image shows the first marker,
determine a first distance from the image sensor to the first marker based on the first image,
determine a first direction of the image sensor relative to the first marker based on the first image, and
change at least one calibration parameter of an eye tracking algorithm used with the eye tracking device based on the first distance and the first direction.

2. The wearable augmented reality or virtual reality headset of claim 1 , wherein the first marker is on a surface of the first lens or in the first lens.

3. The wearable augmented reality or virtual reality headset of claim 1 , wherein the first marker is on a surface of the first lens, wherein the surface substantially faces the image sensor.

4. The wearable augmented reality or virtual reality headset of claim 1 , wherein the headset or the eye tracking device further comprises a second lens comprising a second marker.

5. The wearable augmented reality or virtual reality headset of claim 4 , wherein the second lens comprises a Fresnel lens.

6. The wearable augmented reality or virtual reality headset of claim 5 , wherein the second marker is located at one layer of the Fresnel lens.

7. The wearable augmented reality or virtual reality headset of claim 4 , wherein:
the first image further shows the second marker; and
the one or more processors are further configured to:
determine a second distance from the image sensor to the second marker based on the first image, and
change at least one calibration parameter of the eye tracking algorithm based on the second distance.

8. The wearable augmented reality or virtual reality headset of claim 4 , wherein the one or more processors are further configured to:
receive a second image from the image sensor, wherein the second image shows the second marker;
determine a second distance from the image sensor to the second marker based on the second image; and
change at least one calibration parameter of the eye tracking algorithm based on the second distance.

9. The wearable augmented reality or virtual reality headset of claim 1 , wherein changing the at least one calibration parameter is further based on the first distance being different than a second distance, wherein the second distance is a previously measured and stored distance.

10. The wearable augmented reality or virtual reality headset of claim 1 , wherein changing the at least one calibration parameter based on the first distance comprises:
determining a second distance from the image sensor to the first lens based on the first distance and a known location of the first marker at the first lens; and
changing the at least one calibration parameter of the eye tracking algorithm used with the eye tracking device based on the second distance.

11. The wearable augmented reality or virtual reality headset of claim 1 , wherein at least one of the at least one calibration parameter is selected from a group consisting of:
an imagine parameter;
a focal length;
a distortion parameter; and
an illumination parameter.

12. The wearable augmented reality or virtual reality headset of claim 1 , wherein the first marker comprises a selection from a group consisting of:
a cross mark;
a dot pattern;
a dot;
a line; and
a geometric shape.

13. The wearable augmented reality or virtual reality headset of claim 12 , wherein the first marker comprises a lighted element.

14. The wearable augmented reality or virtual reality headset of claim 1 , wherein:
the headset further comprises a hot mirror comprising a second marker;
one of the first image or a second image received from the image sensor shows the second marker; and
the one or more processors are further configured to:
determine a second distance from the image sensor to the second marker based on the one of the first image or the second image that is showing the second marker, and
change at least one calibration parameter of the eye tracking algorithm used with the eye tracking device based on the second distance.

15. A wearable augmented reality or virtual reality headset wherein the headset comprises:
an eye tracking device comprising:
an image sensor, and
a lens; and

one or more processors configured to at least:
receive image data from the image sensor, wherein the image data corresponds to an image as observed through the lens,
determine a level or a pattern of pincushion distortion in the image based on the image data, and
change at least one calibration parameter of an eye tracking algorithm used with the eye tracking device based on the level or the pattern of pincushion distortion.

16. The wearable augmented reality or virtual reality headset of claim 15 , wherein changing the at least one calibration parameter based on the level or the pattern of pincushion distortion comprises:
determining a relative position or relative direction of the image sensor based on a level or a pattern of pincushion distortion; and
changing the at least one calibration parameter of the eye tracking algorithm used with the eye tracking device based on the relative position or the relative direction of the image sensor.

17. The wearable augmented reality or virtual reality headset of claim 15 , wherein:
the headset further comprises a motion sensor; and
the at least one processor is further configured to:
receive a signal from the motion sensor,
determine, based on the signal, that the headset has accelerated or decelerated beyond a threshold amount, and
initiate at least one action in response to determining that the apparatus headset has accelerated or decelerated beyond the threshold amount.

18. The wearable augmented reality or virtual reality headset of claim 17 , wherein the at least one action comprises at least one selection from a group consisting of:
alerting a user of the apparatus; and
changing at least one calibration parameter of the eye tracking algorithm.

Assignees

• Tobii Ab

Inventors

• Andersson Jonas
• Clausen Anders
• Hainzl Richard
• Kingbäck Anders
• Olin Simon
• Ryan Mark
• Tornéus Daniel
• Nutti Björn
• Sundberg Torbjörn
• Tidbeck Catarina
• Biedert Ralf
• Blomqvist Niklas
• Rådell Dennis
• Thunström Robin

Classifications

• G02B2027/0138

  comprising image capture systems, e.g. camera · CPC title

• G06T2207/30244

  Camera pose · CPC title

• H04N17/002 Primary

  for television cameras · CPC title

• G06V10/141

  Control of illumination · CPC title

• G02B27/0093 Primary

  with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking · CPC title