Face and eye tracking using facial sensors within a head-mounted display

Patent metadata

Field	Value
Publication number	US-2017352183-A1
Application number	US-201615172473-A
Country	US
Kind code	A1
Filing date	Jun 3, 2016
Priority date	Jun 3, 2016
Publication date	Dec 7, 2017
Grant date	—

Abstract

A head mounted display (HMD) in a VR system includes sensors for tracking the eyes and face of a user wearing the HMD. The VR system records calibration attributes such as landmarks of the face of the user. Light sources illuminate portions of the user's face covered by the HMD. In conjunction, facial sensors capture facial data. The VR system analyzes the facial data to determine the orientation of planar sections of the illuminated portions of face. The VR system aggregates planar sections of the face and maps the planar sections to landmarks of the face to generate a facial animation of the user, which can also include eye orientation information. The facial animation is represented as a virtual avatar and presented to the user.

First claim

1 . A head mounted display (HMD) comprising:
a display element configured to display content to a user wearing the HMD; an optics block configured to direct light from the display element to an exit pupil of the HMD; a plurality of light sources positioned at discrete locations around the optics block, the plurality of light sources configured to illuminate portions of a face, inside the HMD, of the user; a facial sensor configured to capture one or more facial data of a portion of the face illuminated by one or more of the plurality of light sources; and a controller configured to:
receive a plurality of captured facial data from the facial sensor;
identify a plurality of surfaces of the portion of the face based at least in part on the plurality of captured facial data;
retrieve a plurality of landmarks each indicating a position of a different feature of the face, the plurality of landmarks determined based on calibration attributes corresponding to a plurality of different facial expressions performed by the user for a calibration process;
map the plurality of surfaces to one of the plurality of landmarks; and
generate facial animation information describing the portion of the face of the user based at least in part on the plurality of captured facial data and the mapping.

2 . The HMD of claim 1 , wherein the controller is further configured to:
provide instructions to the user to perform the plurality of different facial expressions for the calibration process; generate the calibration attributes responsive to the user performing at least one of the plurality of different facial expressions; and identify the plurality of landmarks based at least in part on the calibration attributes.

3 . The HMD of claim 1 , wherein the display element is configured to display an avatar to the user, and the avatar's face is based on the facial animation information.

4 . The HMD of claim 1 , wherein the controller is further configured to:
identify another plurality of surfaces of the portion of the face based at least in part on the plurality of captured facial data; map the other plurality of surfaces to a different one of the plurality of landmarks.

5 . The HMD of claim 1 , wherein the plurality of captured facial data describe frames of an image, the image comprising a plurality of pixels, each pixel associated with a coordinate (x, y) location of the image, and wherein the controller is further configured to:
identify, for each coordinate location of the image, a planar section corresponding to the brightest pixel value at the coordinate location; identify a light source position corresponding to each of the identified planar sections; generate a virtual surface of the portion of the face based at least in part on the identified planar sections and the identified light source positions; and wherein generating the facial animation information is further based on the virtual surface.

6 . The HMD of claim 1 , wherein the controller is further configured to:
provide instructions to the plurality of light sources to illuminate the portions of the face; and provide instructions to the facial sensor, coordinated with the instructions to the plurality of light sources, to capture the one or more facial data.

7 . The HMD of claim 6 , wherein the plurality of light sources are positioned in a ring arrangement around the optics block, and wherein the instructions provided to the plurality of light sources are coordinated such that only one light source of the plurality of light sources illuminates the portions of the face at any given time.

8 . The HMD of claim 1 , wherein the controller is further configured to:
receive specular reflection information from the plurality of facial data; and identify a position of an eye of the user based at least in part on the specular reflection information.

9 . The HMD of claim 8 , wherein the facial animation information is further based on the position of the eye of the user.

10 . A head mounted display (HMD) comprising:
a display element configured to display content to a user wearing the HMD; a plurality of light sources positioned at discrete locations outside a line of sight of the user, the plurality of light sources configured to illuminate portions of a face, inside the HMD, of the user; a facial sensor configured to capture one or more facial data of a portion of the face illuminated by one or more of the plurality of light sources; and a controller configured to:
receive a plurality of captured facial data from the facial sensor;
identify a plurality of surfaces of the portion of the face based at least in part on the plurality of captured facial data;
retrieve a plurality of landmarks each indicating a position of a different feature of the face, the plurality of landmarks determined based on calibration attributes corresponding to a plurality of different facial expressions performed by the user for a calibration process;
map the plurality of surfaces to one of the plurality of landmarks; and
generate facial animation information describing the portion of the face of the user based at least in part on the plurality of captured facial data and the mapping.

11 . The HMD of claim 10 , wherein the controller is further configured to:
provide instructions to the user to perform the plurality of different facial expressions for the calibration process; generate the calibration attributes responsive to the user performing at least one of the plurality of different facial expressions; and identify the plurality of landmarks based at least in part on the calibration attributes.

12 . The HMD of claim 10 , wherein the display element is configured to display an avatar to the user, and the avatar's face is based on the facial animation information.

13 . The HMD of claim 10 , wherein the controller is further configured to:
identify another plurality of surfaces of the portion of the face based at least in part on the plurality of captured facial data; map the other plurality of surfaces to a different one of the plurality of landmarks.

14 . The HMD of claim 10 , wherein the plurality of captured facial data describe frames of an image, the image comprising a plurality of pixels, each pixel associated with a coordinate (x, y) location of the image, and wherein the controller is further configured to:
identify, for each coordinate location of the image, a planar section corresponding to the brightest pixel value at the coordinate location; identify a light source position corresponding to each of the identified planar sections; generate a virtual surface of the portion of the face based at least in part on the identified planar sections and the identified light source positions; and wherein generating the facial animation information is further based on the virtual surface.

15 . The HMD of claim 10 , wherein the controller is further configured to:
provide instructions to the plurality of light sources to illuminate the portions of the face; and provide instructions to the facial sensor, coordinated with the instructions to the plurality of light sources, to capture the one or more facial data.

16 . The HMD of claim 15 , wherein the plurality of light sources are positioned in a ring arrangement around the optics block, and wherein the instructions provided to the plurality of light sources are coordinated such that only one light source of the plurality of light sources illuminates the portion of the face at any given time.

17 . The HMD of claim 10 , wherein the controller

Assignees

• Oculus Vr Llc

Inventors

• Katz Dov
• Toksvig Michael John
• Wang Ziheng
• Omernick Timothy Paul
• Herndon Torin Ross

Classifications

• H04N23/56

  provided with illuminating means · CPC title

• H04N5/2256

  Electricity · mapped topic

• G06T19/006 Primary

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G02B27/0172

  characterised by optical features · CPC title

• G02B2027/0187

  slaved to motion of at least a part of the body of the user, e.g. head, eye · CPC title