Personalized neural network for eye tracking

Patent metadata

Field	Value
Publication number	US-12488488-B2
Application number	US-202117221250-A
Country	US
Kind code	B2
Filing date	Apr 2, 2021
Priority date	Sep 20, 2017
Publication date	Dec 2, 2025
Grant date	Dec 2, 2025

Abstract

Disclosed herein is a wearable display system for capturing retraining eye images of an eye of a user for retraining a neural network for eye tracking. The system captures retraining eye images using an image capture device when user interface (UI) events occur with respect to UI devices displayed at display locations of a display. The system can generate a retraining set comprising the retraining eye images and eye poses of the eye of the user in the retraining eye images (e.g., related to the display locations of the UI devices) and obtain a retrained neural network that is retrained using the retraining set.

First claim

What is claimed is:

1 . A computing system comprising:
a display device; a non-transitory computer-readable storage medium configured to store software instructions; a hardware processor configured to execute the software instructions to cause the computing system to:
capture one or more first images of an eye of a user during or immediately after a first user interface event in which the user activates or deactivates a virtual button of a virtual remote control in a first position, the first images reflecting eye poses of the user which are associated with a particular first portion of a user interface rendered as virtual content;
capture one or more second images of an eye of a user during or immediately after a second user interface event in which the user activates or deactivates the virtual button of the virtual remote control in a second position, the second images reflecting eye poses of the user which are associated with a particular second portion of a user interface, different than the particular first portion, rendered as virtual content;
cause update, based on the obtained first and second images as a set of retraining eye images, of a machine learning model configured to output an eye pose based on an input image related to the particular portion of the user interface, wherein the eye pose indicates a plurality of angular parameters relative to a natural resting direction of the eye and wherein the angular parameters indicate an azimuthal deflection and a zenithal deflection; and
identify, during operation of the computing system, a particular eye pose of the user via applying the updated machine learning model to an input image.

2 . The computing system of claim 1 , wherein the particular portion of the user interface corresponds to a location of the user interface event.

3 . The computing system of claim 1 , wherein the computing system is further configured to: transmit the one or more images and the associated particular portion of the user interface to a remote server configured to update the neural network.

4 . The computing system of claim 1 , wherein the machine learning model is updated to personalize the machine learning model to the user at or proximate to when the one or more images of the eye of the user are obtained during or immediately after the user interface event.

5 . The computing system of claim 1 , wherein the eye poses are determined based on the location of the particular portion and the location of the eye in the one or more images.

6 . The computing system of claim 1 , wherein the computing system comprises a wearable augmented reality headset and the user interface is rendered in a three-dimensional environment, and wherein the input image is obtained via an inward-facing camera.

7 . A computerized method, performed by a computing system having one or
more hardware computer processors and one or more non-transitory computer readable storage device storing software instructions executable by the computing system to perform the computerized method comprising: capturing one or more first images of an eye of a user during or immediately after a first user interface event in which the user activates or deactivates a virtual button of a virtual remote control, the first images reflecting eye poses of the user which are associated with a particular first portion of a user interface rendered as virtual content; capturing one or more additional images of the eye of the user during or immediately after respective further user interface events in which the user activates or deactivates the virtual button of the virtual remote control, the additional images reflecting eye poses of the user which are associated with respective particular further portions of the user interface rendered as virtual content, wherein the first images and the particular first portion and the additional images and the respective particular further portions form a retraining set with retraining input data and corresponding retraining target output data with an eye pose of the eye of the user in each eye image of the eye images related to a display location of the virtual button with respect to the eye image; determining a distribution probability of the virtual button in a first eye pose region of a plurality of eye pose regions according to a probability distribution function; and generating the retraining input data comprising the retraining eye image at an inclusion probability related to the distribution probability of display locations of the virtual button; causing update, based on the obtained images, of a machine learning model configured to output an eye pose based on an input image, wherein the eye pose indicates a plurality of angular parameters relative to a natural resting direction of the eye and wherein the angular parameters indicate an azimuthal deflection and a zenithal deflection; and identifying, during operation of the computing system, a particular eye pose of the user via applying the updated machine learning model to an input image.

8 . The method of claim 7 , wherein the particular portion of the user interface corresponds to the location of a user interface event.

9 . The method of claim 7 , further comprising:
transmitting the one or more images and the associated particular portion of the user interface to a remote server configured to update the neural network.

10 . The method of claim 7 , wherein the machine learning model is updated to personalize the machine learning model to the user, and wherein weights of the updated machine learning model are set to initial weights corresponding to the weights prior to updating the machine learning model.

11 . The method of claim 7 , wherein the eye poses are determined based on the location of the particular portion and the location of the eye in the one or more images.

12 . A non-transitory computer readable medium having software instructions stored thereon, the software instructions executable by a hardware computer processor to cause a computing system to perform operations comprising:
capturing one or more first images of an eye of a user during or immediately after a first user interface event in which the user activates or deactivates a virtual button of a virtual remote control, the one or more first images reflecting eye poses of the user which are associated with a particular first portion of a user interface rendered as virtual content, wherein the virtual remote control during the first user interface event has a primary function other than capturing the one or more first images of the user; capture one or more second images of an eye of a user during or immediately after a second user interface event in which the user activates or deactivates the virtual button of the virtual remote control, the one or more second images reflecting eye poses of the user which are associated with a particular second portion of a user interface, different than the particular first portion, rendered as virtual content; causing update, based on the obtained one or more first and second images as a set of retraining eye images, of a machine learning model configured to output an eye pose based on an input image, wherein the eye pose indicates a plurality of angular parameters relative to a natural resting direction of the eye and wherein the angular parameters indicate an azimuthal deflection and a zenithal deflection; and identifying, during operation of the computing system, a particular eye pose of the user via applying the updated machine learning model to an input image.

13 . The non-transitory storage media of claim 12 , wherein the particular first por

Assignees

• Magic Leap Inc

Inventors

• Kaehler Adrian
• Lee Douglas Bertram
• Badrinarayanan Vijay

Classifications

• G06N3/08

  Learning methods · CPC title

• G02B27/0093 Primary

  with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking · CPC title

• G06F3/011

  Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00) · CPC title

• G06T2207/10048

  Infrared image · CPC title

• G06F1/163

  Wearable computers, e.g. on a belt · CPC title