Personalized neural network for eye tracking

Patent metadata

Field	Value
Publication number	US-10977820-B2
Application number	US-202016880752-A
Country	US
Kind code	B2
Filing date	May 21, 2020
Priority date	Sep 20, 2017
Publication date	Apr 13, 2021
Grant date	Apr 13, 2021

Abstract

Disclosed herein is a wearable display system for capturing retraining eye images of an eye of a user for retraining a neural network for eye tracking. The system captures retraining eye images using an image capture device when user interface (UI) events occur with respect to UI devices displayed at display locations of a display. The system can generate a retraining set comprising the retraining eye images and eye poses of the eye of the user in the retraining eye images (e.g., related to the display locations of the UI devices) and obtain a retrained neural network that is retrained using the retraining set.

First claim

What is claimed is:

1. A computing system comprising:
a display device;
a non-transitory computer-readable storage medium configured to store software instructions;
a hardware processor configured to execute the software instructions to cause the computing system to:
detect a user interface event based on a user interaction of a user with a particular portion of a user interface rendered as virtual content within the user's field of view via the display device;
identify one or more images of an eye of the user acquired during or immediately after the user interface event, the images reflecting eye poses of the user which are directed to the particular portion of the user interface; and
initiate update of a neural network configured to detect eye pose of users based on eye images of the user, wherein the neural network is updated based on the one or more images reflecting eye poses of the user and the particular portion of the user interface,
wherein the updated neural network is personalized for detecting eye pose of the user.

2. The computing system of claim 1 , wherein the neural network is trained to detect eye pose of users based on a plurality of training images from individuals other than the user.

3. The computing system of claim 1 , wherein the user interface event comprises selection of a user interface element.

4. The computing system of claim 3 , wherein the particular portion of the user interface comprises a location of the user interface element.

5. The computing system of claim 1 , wherein said initiating update of the neural network comprises:
transmitting the one or more images of the eye of the user and the associated particular portion of the user interface to a remote server configured to update the neural network.

6. The computing system of claim 5 , further comprising:
receiving, from the remote server, the updated neural network personalized for detecting eye pose of the user.

7. The computing system of claim 1 , wherein the software instructions are further configured to cause the computing system to:
detect additional user interface events based on user interactions with particular portions of one or more user interfaces rendered on the display device;
for each detected additional user interface event, identify one or more images of the eye of the user acquired during or immediately after the additional user interface event; and
periodically optimizing the updated neural network based on one or more sets of additional user interface events and corresponding one or more images of the eye of the user,
wherein the optimized neural network is further personalized for detecting eye pose of the user.

8. The computing system of claim 1 , wherein prior to updating of the neural network, the neural network is not personalized to the user.

9. The computing system of claim 1 , wherein the computing system comprises a wearable augmented reality headset and the user interface is rendered in a three-dimensional environment.

10. The computing system of claim 9 , wherein the display device comprises a left display and a right display.

11. A computerized method, performed by a computing system having one or more hardware computer processors and one or more non-transitory computer readable storage device storing software instructions executable by the computing system to perform the computerized method comprising:
detecting a user interface event based on a user interaction with a particular portion of a user interface rendered as virtual content within a user's field of view via a display device;
identifying one or more images of an eye of the user acquired during or immediately after the user interface event, the images reflecting eye poses which are directed to the particular portion of the user interface; and
initiating update of a neural network configured to detect eye pose of users based on eye images of the user, wherein the neural network is updated based on the one or more images reflecting eye poses of the user and the particular portion of the user interface,
wherein the updated neural network is personalized for detecting eye pose of the user.

12. The computerized method of claim 11 , wherein the neural network is trained to detect eye pose of users based on a plurality of training images from individuals other than the user.

13. The computerized method of claim 11 , wherein the user interface event comprises selection of a user interface element.

14. The computerized method of claim 13 , wherein the particular portion of the user interface comprises a location of the user interface element.

15. The computerized method of claim 11 , wherein said initiating update of the neural network comprises:
transmitting the one or more images of the eye of the user and the associated particular portion of the user interface to a remote server configured to update the neural network.

16. A non-transitory computer readable medium having software instructions stored thereon, the software instructions executable by a hardware computer processor to cause a computing system to perform operations comprising:
detecting a user interface event based on a user interaction with a particular portion of a user interface rendered as virtual content within the user's field of view via a display device;
identifying one or more images of an eye of the user acquired during or immediately after the user interface event, the images reflecting eye poses of the user which are directed to the particular portion of the user interface; and
initiating update of a neural network configured to detect eye pose of users based on eye images of the user, wherein the neural network is updated based on the one or more images reflecting eye poses of the user and the particular portion of the user interface,
wherein the updated neural network is personalized for detecting eye pose of the user.

17. The computerized method of claim 16 , wherein the neural network is trained to detect eye pose of users based on a plurality of training images from individuals other than the user.

18. The computerized method of claim 16 , wherein the user interface event comprises selection of a user interface element.

19. The computerized method of claim 18 , wherein the particular portion of the user interface comprises a location of the user interface element.

20. The computerized method of claim 16 , wherein said initiating update of the neural network comprises:
transmitting the one or more images of the eye of the user and the associated particular portion of the user interface to a remote server configured to update the neural network.

Assignees

• Magic Leap Inc

Inventors

• Kaehler Adrian
• Lee Douglas Bertram
• Badrinarayanan Vijay

Classifications

• G06N3/08

  Learning methods · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06T2207/30201

  Face · CPC title

• G06T2207/30041

  Eye; Retina; Ophthalmic · CPC title