Deep learning for three dimensional (3D) gaze prediction

Patent metadata

Field	Value
Publication number	US-11301677-B2
Application number	US-202016901377-A
Country	US
Kind code	B2
Filing date	Jun 15, 2020
Priority date	Jun 14, 2019
Publication date	Apr 12, 2022
Grant date	Apr 12, 2022

Abstract

There is disclosed a computer implemented eye tracking system and corresponding method and computer readable storage medium, for detecting three dimensional, 3D, gaze, by obtaining at least one head pose parameter using a head pose prediction algorithm, the head pose parameter(s) comprising one or more of a head position, pitch, yaw, or roll; and to input the at least one head pose parameter along with at least one image of a user's eye, generated from a 2D image captured using an image sensor associated with the eye tracking system, into a neural network configured to generate 3D gaze information based on the at least one head pose parameter and the at least one eye image.

First claim

The invention claimed is:

1. A computer implemented method for detecting three dimensional, 3D, gaze, using an eye tracking system the computer-implemented method comprising:
obtaining, by a head pose prediction algorithm, at least one head pose parameter, the at least one head pose parameter comprising one or more of: a head position, pitch, yaw, or roll;
generating, by the eye tracking system, a first eye image comprising a first eye of a user based on a first two dimensional, 2D, image, wherein the first 2D image is generated by an image sensor associated with the eye tracking system;
inputting, by the eye tracking system to a neural network, NN, the first eye image, wherein the NN is configured to convert the first eye image into a first feature vector;
inputting, by the eye tracking system to a concatenation layer comprised in the NN, the obtained at least one head pose parameter and the first feature vector, wherein the concatenation layer is configured to generate a resulting feature vector, by concatenating the obtained at least one head pose parameter and the first feature vector;
inputting, by the eye tracking system to a fully connected, FC, module comprised in the neural network, the generated resulting feature vector, wherein the FC module of the neural network is configured to predict a 2D gaze origin of the first eye of the user in the first eye image, and a 2D gaze direction of the first eye of the user in the first eye image based on the resulting feature vector; and
generating, by the eye tracking system, 3D gaze information for the first eye of the user based on the 2D gaze origin and the 2D gaze direction.

2. The computer-implemented method of claim 1 , wherein the FC module of the neural network is further configured to predict a distance correction based on the resulting feature vector; wherein the method further comprises generating, by the eye tracking system, a corrected distance between the first eye of the user and the image sensor by at least updating an estimated distance based on the distance correction; and wherein generating, by the eye tracking system, 3D gaze information for the first eye of the user is further based on the corrected distance.

3. The computer-implemented method of claim 2 , further comprising:
determining, by the eye tracking system, the estimated distance between the image sensor and the first eye of the user based on the first 2D image; and
estimating, by the eye tracking system, a position of the first eye of the user in a 3D space based on the corrected distance and on a position of the image sensor in the 3D space, wherein the 3D gaze information comprises the position of the first eye of the user in the 3D space.

4. The computer-implemented method of claim 1 , further comprising estimating, by the eye tracking system, a 3D gaze direction from a position of the first eye of the user in a 3D space based on the 2D gaze origin and the 2D gaze direction, wherein the 3D gaze information comprises the 3D gaze direction.

5. The computer-implemented method of claim 1 , wherein generating the first eye image comprises normalizing the first 2D image to generate a normalized image and cropping the normalized image around the first eye of the user.

6. The computer-implemented method of claim 1 , wherein the first 2D image further shows a second eye of the user, and wherein the method further comprises:
generating, by the eye tracking system, a second eye image comprising the second eye of the user, based on the first 2D image; and
inputting, by the eye tracking system to the fully connected module comprised in the neural network, the second eye image, wherein the fully connected module of the neural network is further configured to convert the second eye image into a second feature vector,
inputting, by the eye tracking system to a concatenation layer comprised in the NN, also the second feature vector, wherein the concatenation layer is configured to generate the resulting feature vector, by concatenating the obtained at least one head pose parameter, the first feature vector and the second feature vector.

7. The computer-implemented method of claim 6 , further comprising:
generating, by the eye tracking system, a second position of the second eye of the user in a 3D space and a second gaze direction from the second position in the 3D space based on a corrected distance, and wherein the 3D gaze information comprises the second position and the second gaze direction.

8. The computer-implemented method of claim 7 , wherein the second eye image is a mirrored image around the second eye of the user.

9. The computer-implemented method of claim 5 , wherein the first eye image and the second eye image are generated at a first image resolution, based on a first predefined distance associated with projecting the first eye image and the second eye image in a 3D gaze space.

10. The computer-implemented method of claim 6 , wherein the neural network is further configured to predict a second 2D gaze origin and a second 2D gaze direction of the second eye of the user in the second eye image based on the first eye image and the second eye image.

11. The computer-implemented method of claim 6 , wherein the neural network is further configured to predict a second 2D gaze origin and a second 2D gaze direction of the second eye in the second eye image based on the second eye image and independently of the first eye image.

12. An eye tracking system for detecting three dimensional, 3D, gaze, comprising:
an image sensor;
a processor; and
a memory storing computer-readable instructions that, upon execution by the processor, cause the eye tracking system to perform operations comprising:
obtaining, by a head pose prediction algorithm, at least one head pose parameter, the at least one head pose parameter comprising one or more of: a head position, pitch, yaw, or roll;
generating, by the eye tracking system, a first eye image comprising a first eye of a user based on a first two dimensional, 2D, image, wherein the first 2D image is generated by an image sensor;
inputting, by the eye tracking system to a neural network, NN, the first eye image, wherein the NN is configured to convert the first eye image into a first feature vector;
inputting, by the eye tracking system to a concatenation layer comprised in the NN, the obtained at least one head pose parameter and the first feature vector, wherein the concatenation layer is configured to generate a resulting feature vector, by concatenating the obtained at least one head pose parameter and the first feature vector;
inputting, by the eye tracking system to a fully connected, FC, module comprised in the neural network, the generated resulting feature vector, wherein the FC module of the neural network is configured to predict a 2D gaze origin of the first eye of the user in the first eye image, and a 2D gaze direction of the first eye of the user in the first eye image based on the resulting feature vector; and
generating, by the eye tracking system, 3D gaze information for the first eye of the user based on the 2D gaze origin and the 2D gaze direction.

13. The eye tracking system of claim 12 , wherein the FC module of the neural network is further configured to predict a distance correction based on the resulting feature vector and wherein the memory is further configured to store computer-readable instructions that, upon execution by the processor, cause the eye tracking system to perform operations comprising:
generating, by the eye tracking system, a corrected distance between the first eye of the user and the image sensor by at least updating an estimated distance base

Assignees

• Tobii Ab
• Tobil AB

Inventors

• Molin David
• Martini Tommaso
• Gordon Maria
• Davies Alexander
• Danielsson Oscar

Classifications

• G06V40/18

  Eye characteristics, e.g. of the iris · CPC title

• G06T7/73 Primary

  using feature-based methods · CPC title

• G06V10/764

  using classification, e.g. of video objects · CPC title

• G06V40/19

  Sensors therefor · CPC title

• G06F18/21

  Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation · CPC title