Personalized neural network for eye tracking

Patent metadata

Field	Value
Publication number	US-10719951-B2
Application number	US-201816134600-A
Country	US
Kind code	B2
Filing date	Sep 18, 2018
Priority date	Sep 20, 2017
Publication date	Jul 21, 2020
Grant date	Jul 21, 2020

Abstract

Disclosed herein is a wearable display system for capturing retraining eye images of an eye of a user for retraining a neural network for eye tracking. The system captures retraining eye images using an image capture device when user interface (UI) events occur with respect to UI devices displayed at display locations of a display. The system can generate a retraining set comprising the retraining eye images and eye poses of the eye of the user in the retraining eye images (e.g., related to the display locations of the UI devices) and obtain a retrained neural network that is retrained using the retraining set.

First claim

What is claimed is:

1. A wearable display system comprising:
an image capture device configured to capture a plurality of retraining eye images of an eye of a user;
a display;
non-transitory computer-readable storage medium configured to store:
the plurality of retraining eye images, and
a neural network for eye tracking; and

a hardware processor in communication with the image capture device, the display, and the non-transitory computer-readable storage medium, the hardware processor programmed by the executable instructions to:
receive the plurality of retraining eye images captured by the image capture device,
wherein a retraining eye image of the plurality of retraining eye images is captured by the image capture device when a user interface (UI) event, with respect to a UI device shown to a user at a display location of the display, occurs;

generate a retraining set comprising retraining input data and corresponding retraining target output data,
wherein the retraining input data comprises the retraining eye images, and
wherein the corresponding retraining target output data comprises an eye pose of the eye of the user in the retraining eye image related to the display location; and

obtain a retrained neural network that is retrained from a neural network for eye tracking using the retraining set.

2. The wearable display system of claim 1 , wherein to obtain the retrained neural network, the hardware processor is programmed to at least:
retrain the neural network for eye tracking using the retraining set to generate the retrained neural network.

3. The wearable display system of claim 1 , wherein to obtain the retrained neural network, the hardware processor is programmed to at least:
transmit the retraining set to a remote system; and
receive the retrained neural network from the remote system.

4. The wearable display system of claim 3 , wherein the remote system comprises a cloud computing system.

5. The wearable display system of claim 1 , wherein to receive the plurality of retraining eye images of the user, the hardware processor is programmed by the executable instructions to at least:
display the UI device to the user at the display location on the display;
determine an occurrence of the UI event with respect to the UI device; and
receive the retraining eye image from the image capture device.

6. The wearable display system of claim 1 , wherein the hardware processor is further programmed by the executable instructions to:
determine the eye pose of the eye in the retraining eye image using the display location.

7. The wearable display system of claim 6 , wherein the eye pose of the eye in the retraining image comprises the display location.

8. The wearable display system of claim 1 , wherein to receive the plurality of retraining eye images of the user, the hardware processor is programmed by the executable instructions to at least:
generate a second plurality of second retraining eye images based on the retraining eye image; and
determine an eye pose of the eye in a second retraining eye image of the second plurality of second retraining eye images using the display location and a probability distribution function.

9. The wearable display system of claim 1 , wherein to receive the plurality of retraining eye images of the user, the hardware processor is programmed by the executable instructions to at least:
receive a plurality of eye images of the eye of the user from the image capture device,
wherein a first eye image of the plurality of eye images is captured by the user device when the UI event, with respect to the UI device shown to the user at the display location of the display, occurs;

determine a projected display location of the UI device from the display location, backward along a motion of the user prior to the UI event, to a beginning of the motion;
determine the projected display location and a second display location of the UI device in a second eye image of the plurality of eye images captured at the beginning of the motion are with a threshold distance; and
generate the retraining input data comprising eye images of the plurality of eye images from the second eye image to the first eye image,
wherein the corresponding retraining target output data comprises an eye pose of the eye of the user in each eye image of the eye images related to a display location of the UI device in the eye image.

10. The wearable display system of claim 1 , wherein the eye pose of the eye is the display location.

11. The wearable display system of claim 1 , wherein hardware processor is further programmed by the executable instructions to at least: determine the eye pose of the eye using the display location of the UI device.

12. The wearable display system of claim 1 , wherein to generate the retraining set, the hardware processor is programmed by the executable instructions to at least:
determine the eye pose of the eye in the retraining eye image is in a first eye pose region of a plurality of eye pose regions;
determine a distribution probability of the UI device being in the first eye pose region; and
generate the retraining input data comprising the retraining eye image at an inclusion probability related to the distribution probability.

13. The wearable display system of claim 1 , wherein the hardware processor is further programmed by the executable instructions to at least:
train the neural network for eye tracking using a training set comprising training input data and corresponding training target output data,
wherein the training input data comprises a plurality of training eye images of a plurality of users, and
wherein the corresponding training target output data comprises eye poses of eyes of the plurality of users in the training plurality of training eye images.

14. The wearable display system of claim 13 , wherein the retraining input data of the retraining set comprises at least one training eye image of the plurality of training eye images.

15. The wearable display system of claim 13 , wherein the retraining input data of the retraining set comprises no training eye image of the plurality of training eye images.

16. The wearable display system of claim 1 , wherein to retrain the neural network for eye tracking, the hardware processor is programmed by the executable instructions to at least:
initialize weights of the retrained neural network with weights of the neural network.

17. The wearable display system of claim 1 , wherein the hardware processor is programmed by the executable instructions to cause the user device to:
receive an eye image the user from the image capture device; and
determine an eye pose of the user in the eye image using the retrained neural network.

18. A system for retraining a neural network for eye tracking, the system comprising:
computer-readable memory storing executable instructions; and
one or more processors programmed by the executable instructions to at least:
receive a plurality of retraining eye images of an eye of a user,
wherein a retraining eye image of the plurality of retraining eye images is captured when a user interface (UI) event, with respect to a UI device shown to a user at a display location of a user device, occurs;

generating a retraining set comprising retraining input data and corresponding retraining target output data,
wherein the retraining input data comprises the retraining eye images, and
wherein the corresponding retraining target output data comprises an eye pose of the eye of the user in the retrai

Assignees

• Magic Leap Inc

Inventors

• Kaehler Adrian
• Lee Douglas Bertram
• Badrinarayanan Vijay

Classifications

• G06N3/08

  Learning methods · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06T2207/30201

  Face · CPC title

• G06T2207/30041

  Eye; Retina; Ophthalmic · CPC title