Determining fixation of a user's eyes from images of portions of the user's face enclosed by a head mounted display

Patent metadata

Field	Value
Publication number	US-10795436-B2
Application number	US-201916683014-A
Country	US
Kind code	B2
Filing date	Nov 13, 2019
Priority date	Jun 1, 2018
Publication date	Oct 6, 2020
Grant date	Oct 6, 2020

Abstract

A virtual reality (VR) or augmented reality (AR) head mounted display (HMD) includes multiple image capture devices positioned within the HMD to capture portions of a face of a user wearing the HMD. Images from an image capture device include a user's eye, while additional images from another image capture device include the user's other eye. The images and the additional images are provided to a controller, which applies a trained model to the images and the additional images to generate a vector identifying a position of the user's head and positions of the user's eye and fixation of each of the user's eyes. Additionally, illumination sources illuminating portions of the user's face include in the images and in the additional images are configured when the user wears the HMD to prevent over-saturation or under-saturation of the images and the additional images.

First claim

What is claimed is:

1. A device comprising:
a left camera enclosed by a rigid body and configured to capture images of a portion of a user's face enclosed by the rigid body including the user's left eye;
a left illumination source enclosed by the rigid body and configured to emit light illuminating the portion of the user's face, the left illumination source comprising a plurality of light emitting diodes (LEDs) positioned around a circumference of a lens of the left camera;
a right camera enclosed by the rigid body and configured to capture additional images of an additional portion of the user's face enclosed by the rigid body including the user's right eye;
a right illumination source enclosed by the rigid body and configured to emit light illuminating the additional portion of the user's face; and
a controller configured to:
modify light emitted by the left illumination source based on one or more of the images; and
modify light emitted by the right illumination source based on one or more of the additional images.

2. The device of claim 1 , wherein the controller is further configured to:
generate a vector indicating a fixation of the user's left eye and a fixation of the user's right eye relative to the position of the head of the user by applying a model to the images and to the additional images.

3. The device of claim 2 , wherein the model comprises a trained convolutional neural network.

4. The device of claim 1 , wherein the left illumination source comprises a plurality of light emitting diodes (LEDs), and modify light emitted by the left illumination source based on one or more of the images comprises:
modify light emitted by at least a set of LEDs comprising the left illumination source to minimize a function based on saturation with light of the one or more images captured by the left camera so the function has a minimum value.

5. The device of claim 4 , wherein the right illumination source comprises a plurality of light emitting diodes (LEDs), and modify light emitted by the right illumination source based on one or more of the additional images comprises:
modifying light emitted by at least a set of LEDs comprising the right illumination source to minimize a function based on saturation with light of the one or more additional images captured by the right camera so the function has the minimum value.

6. The device of claim 1 , wherein modify light emitted by the left illumination source based on one or more images captured by the left illumination source comprises:
modify light emitted by one or more portions of the left illumination source based on one or more images captured by the left camera in response to receiving an indication the device is initially worn by the user.

7. The device of claim 6 , wherein modify light emitted by the right illumination source based on one or more images captured by the right illumination source comprises:
modify light emitted by one or more portions of the right illumination source based on one or more images captured by the right camera in response to receiving the indication the device is initially worn by the user.

8. The device of claim 1 , wherein the right illumination source comprises an additional plurality of light emitting diodes (LEDs) positioned around a circumference of a lens of the right camera.

9. A method comprising:
capturing calibration images of a portion of a user's face enclosed by a head mounted display (HMD) via a left image capture device included in the HMD, the portion of the user's face including a left eye of the user;
modifying light emitted by one or more portions of a left illumination source onto the portion of the user's face enclosed by the HMD based on the calibration images, the left illumination source comprising a plurality of light emitting diodes (LEDs) positioned around a circumference of a lens of the left image capture device;
capturing additional calibration images of a portion of a user's face enclosed by a head mounted display (HMD) via a right image capture device included in the HMD, the additional portion of the user's face including a right eye of the user;
modifying light emitted by one or more portions of a right illumination source onto the additional portion of the user's face enclosed by the HMD based on the additional calibration images by modifying light emitted by a LED positioned around the circumference of the lens of the left image capture device to minimize a function based on saturation with light of the one or more calibration images so the function has a minimum value;
capturing images of the portion of a user's face enclosed by a head mounted display (HMD) via the left image capture device included in the HMD, the portion of the user's face including the left eye of the user;
capturing additional images of the additional portion of the user's face enclosed by the head mounted display via the right image capture device included in the HMD, the additional portion of the user's face including the right eye of the user;
applying a model to the images and to the additional images, the model trained based on previously captured images including portions of users' faces including left eyes and previously captured images including portions of users' faces including right eyes; and
generating a vector indicating a fixation of the user's left eye and a fixation of the user's right eye relative to a position of the head of the user from application of the model to the images and to the additional images.

10. The method of claim 9 , wherein the right illumination source comprises a plurality of additional light emitting diodes (LEDs) positioned around a circumference of a lens of the right image capture device and modifying light emitted by one or more portions of the right illumination source onto the additional portion of the user's face enclosed by the HMD based on one or more additional calibration images captured by the right image capture device comprises:
modifying light emitted by an additional LED positioned around the circumference of the lens of the right image capture device to minimize the function based on saturation with light of the one or more additional calibration images so the function has the minimum value.

11. The method of claim 9 , wherein the model comprises a trained convolutional neural network.

12. The method of claim 9 , wherein capturing images of the portion of a user's face enclosed by the HMD via the left image capture device included in the HMD comprises:
capturing the images of the portion of the user's face enclosed by the HMD via the left image capture device included in the HMD after after modifying the light emitted by the one or more portions of the left illumination source.

13. The method of claim 12 , wherein capturing the additional images of the additional portion of the user's face enclosed by the head mounted display via the right image capture device included in the HMD comprises:
capturing the additional images of the portion of the user's face enclosed by the HMD via the right image capture device included in the HMD after modifying the light emitted by the one or more portions of the right illumination source.

14. A computer program product comprising a non-transitory computer readable storage medium having instructions encoded thereon that, when executed by a processor, cause the processor to:
capture calibration images of a portion of a user's face enclosed by a head mounted display (HMD) via a left image capture device included in the HMD, the portion of the user's face including a left eye of the user;
modify light emitted by one or more portions of

Assignees

• Facebook Tech Llc

Inventors

• Wei Shih-En
• Saragih Jason
• Badino Hernan
• Hypes Alexander Trenor
• Shahmohammadi Mohsen
• Wang Dawei
• Perdoch Michal

Classifications

• G06N3/045

  Combinations of networks · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• H04N13/344

  with head-mounted left-right displays · CPC title

• G06N5/046

  Forward inferencing; Production systems · CPC title