Determining fixation of a user's eyes from images of portions of the user's face enclosed by a head mounted display

Patent metadata

Field	Value
Publication number	US-10509467-B1
Application number	US-201815996422-A
Country	US
Kind code	B1
Filing date	Jun 1, 2018
Priority date	Jun 1, 2018
Publication date	Dec 17, 2019
Grant date	Dec 17, 2019

Abstract

A virtual reality (VR) or augmented reality (AR) head mounted display (HMD) includes multiple image capture devices positioned within the HMD to capture portions of a face of a user wearing the HMD. Images from an image capture device include a user's eye, while additional images from another image capture device include the user's other eye. The images and the additional images are provided to a controller, which applies a trained model to the images and the additional images to generate a vector identifying a position of the user's head and positions of the user's eye and fixation of each of the user's eyes. Additionally, illumination sources illuminating portions of the user's face include in the images and in the additional images are configured when the user wears the HMD to prevent over-saturation or under-saturation of the images and the additional images.

First claim

What is claimed is:

1. A head mounted display (HMD) comprising:
a rigid body having a front side, a left side, a right side, a top side, and a bottom side and including a display element configured to display content to a user wearing the HMD and an optics block configured to direct light from the display element to an exit pupil and an additional exit pupil of the HMD;
a left image capture device coupled to an interior surface of the HMD proximate to the left side of the front rigid body and configured to capture images of a portion of the user's face enclosed by the rigid body, the portion of the user's face including a left eye of the user;
a left illumination source positioned proximate to the left image capture device and configured to emit light illuminating the portion of the user's face;
an right image capture device coupled to an interior surface of the HMD proximate to the right side of the front rigid body and proximate to the bottom side of the rigid body and configured to capture images of an additional portion of the user's face enclosed by the rigid body, the additional portion of the user's face including a right eye of the user;
a right illumination source positioned proximate to the right image capture device and configured to emit light illuminating the additional portion of the user's face; and
a controller coupled to the left image capture device and to the right image capture device, the controller configured to:
obtain the images of the portion of the user's face including the left eye of the user from the left image capture device;
obtain the images of the additional portion of the user's face including the right eye of the user from the right image capture device;
generate a vector indicating a fixation of the user's left eye and a fixation of the user's right eye relative to the position of the head of the user by applying a model to the images and to the additional images;
modify content presented by an electronic display included in the HMD based on the vector by increasing a resolution of a segment of content at a location of the electronic display corresponding to the fixation of the user's left eye relative to a resolution of content presented at other locations of the electronic display and increasing a resolution of an additional segment of content at a location of the electronic display corresponding to the fixation of the user's right eye relative to the resolution of content presented at other locations of the electronic display.

2. The HMD of claim 1 , wherein the controller is further configured to:
transmit the vector to a console, the console configured to generate content for presentation by an electronic display included in the HMD based on the fixation of the user's left eye and the fixation of the user's right eye.

3. The HMD of claim 1 , wherein the model comprises a trained convolutional neural network.

4. The HMD of claim 3 , wherein the trained convolutional neural network is trained based on application of a gradient descent process to images of portion of other users' faces including left eyes of the other users obtained during a calibration process identifying a position of the other users' heads when the images of portion of other users' faces including left eyes of the other users were obtained and images of portions of the other users' faces including right eyes of the other users obtained during the calibration process identifying a position of the other users' heads when the images of portion of other users' faces including right eyes of the other users were obtained.

5. The HMD of claim 4 , wherein the trained convolutional neural network is further trained based on application of the gradient descent process to calibration images of the portion of the user's face including the left eye of the user, the calibration images identifying a position of the user's head when the calibration images were captured, and to additional calibration images of the additional portion of the user's face including the right eye of the user, the additional calibration images identifying a position of the user's head when the additional calibration images were captured.

6. The HMD of claim 1 , wherein the controller is further configured to
modify light emitted by the left illumination source based on one or more images captured by the left illumination source; and
modify light emitted by the right illumination source based on one or more images captured by the right illumination source.

7. The HMD of claim 6 , wherein the left illumination source comprises a plurality of light emitting diodes (LEDs), and modify light emitted by the left illumination source based on one or more images captured by the left illumination source comprises:
modifying light emitted by at least a set of LEDs comprising the left illumination source to minimize a function based on saturation of the one or more images captured by the left illumination source.

8. The HMD of claim 7 , wherein the right illumination source comprises a plurality of light emitting diodes (LEDs), and modify light emitted by the right illumination source based on one or more images captured by the right illumination source comprises:
modifying light emitted by at least a set of LEDs comprising the right illumination source to minimize a function based on saturation of the one or more images captured by the right illumination source.

9. The HMD of claim 6 , wherein modify light emitted by the left illumination source based on one or more images captured by the left illumination source comprises:
modify light emitted by one or more portions of the left illumination source based on one or more images captured by the left illumination source in response to receiving an indication the HMD is initially worn by the user.

10. The HMD of claim 9 , wherein modify light emitted by the right illumination source based on one or more images captured by the right illumination source comprises:
modify light emitted by one or more portions of the right illumination source based on one or more images captured by the right illumination source in response to receiving the indication the HMD is initially worn by the user.

11. The HMD of claim 1 , wherein the left illumination source comprises a plurality of light emitting diodes (LEDs) positioned around a circumference of a lens of the left image capture device.

12. The HMD of claim 11 , wherein the right illumination source comprises an additional plurality of light emitting diodes (LEDs) positioned around a circumference of a lens of the right image capture device.

13. A method comprising:
capturing images of a portion of a user's face enclosed by a head mounted display (HMD) via a left image capture device included in the HMD, the portion of the user's face including a left eye of the user;
capturing images of an additional portion of the user's face enclosed by the head mounted display via a right image capture device included in the HMD, the additional portion of the user's face including a right eye of the user;
applying a model to the images and to the additional images, the model trained based on previously captured images including portions of users' faces including left eyes and previously captured images including portions of users' faces including right eyes;
generating a vector indicating a fixation of the user's left eye and a fixation of the user's right eye relative to a position of the head of the user from application of the model to the images and to the additional images; and
modify content presented by an electronic display included in the HMD based on the vector to visually distinguish content at a loc

Assignees

• Facebook Tech Llc

Inventors

• Wei Shih-En
• Saragih Jason
• Badino Hernan
• Hypes Alexander Trenor
• Shahmohammadi Mohsen
• Wang Dawei
• Perdoch Michal

Classifications

• G02B2027/014

  comprising information/image processing systems · CPC title

• G02B2027/0138

  comprising image capture systems, e.g. camera · CPC title

• G02B27/0093

  with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G06F3/04845

  for image manipulation, e.g. dragging, rotation, expansion or change of colour · CPC title