Generating graphical representation of a user's face and body using a monitoring system included on a head mounted display

Patent metadata

Field	Value
Publication number	US-10636193-B1
Application number	US-201816022758-A
Country	US
Kind code	B1
Filing date	Jun 29, 2018
Priority date	Jun 29, 2017
Publication date	Apr 28, 2020
Grant date	Apr 28, 2020

Abstract

A virtual reality (VR) or augmented reality (AR) head mounted display (HMD) includes various image capture devices that capture images of portions of the user's face and body. Through image analysis, points of each portion of the user's face and body are identified from the images and their movement is tracked. The identified points are mapped to a three dimensional model of a face and to a three dimensional model of a body. From the identified points, animation parameters describing positioning of various points of the user's face and body are determined for each captured image. From the animation parameters and transforms mapping the captured images to three dimensions, the three dimensional model of the face and the three dimensional model of the body is altered to render movement of the user's face and body.

First claim

What is claimed is:

1. A method comprising:
obtaining images of portions of a user's face enclosed by a head mounted device (HMD) captured by one or more image capture devices coupled to the HMD;
obtaining images of additional portions of the user's face outside of the HMD captured by one or more additional image capture devices coupled to the HMD;
identifying points corresponding to features of the portions of the user's face within the images of the portions of the user's face and within the images of the additional portions of the user's face;
determining regularization parameters for different portions of the user's face based on the images captured by the one or more image capture devices and captured by the one or more additional image capture devices;
mapping the identified points to a three-dimensional model of a face;
generating a set of animation parameters for each portion of the user's face and for each additional portion of the user's face from identified points within the images captured by the one or more image capture devices and captured by the one or more additional image capture devices;
applying the corresponding determined regularization parameters to the corresponding generated set of animation parameters for each portion of the user's face and for each additional portion of the user's face to generate a modified set of animation parameters for each portion of the user's face and for each additional portion of the user's face;
determining one or more transformations mapping images captured by the one or more image capture devices and captured by the one or more additional image capture devices into three dimensions;
generating a rendering model based on the modified sets of animation parameters and the determined transformations;
repositioning one or more points of the three-dimensional model of the face by applying the rendering model to the three-dimensional model of the face; and
generating content presenting the three-dimensional model of the face with the repositioned one or more points.

2. The method of claim 1 , wherein the features of the portions of the user's face within the images of the portions of the user's face and within the images of the additional portions of the user's face comprises boundaries of one or more parts of the user's face.

3. The method of claim 1 , wherein identifying points corresponding to features of the portions of the user's face within the images of the portions of the user's face and within the images of the additional portions of the user's face comprises:
applying a model trained from various additional users identifying points corresponding to features of faces of each additional user in previously captured images of the additional users' faces while the additional users make different specific facial expressions to the images of the portions of the user's face and the images of the additional portions of the user's face.

4. The method of claim 1 , wherein mapping the identified points to the three-dimensional model of a face comprises:
selecting the three-dimensional model of the face from a stored library of three-dimensional models of faces based on characteristics of the user; and
identifying points on the three-dimensional model corresponding to the identified points.

5. The method of claim 4 , wherein selecting the three-dimensional model of the face from the stored library of three-dimensional models of faces based on characteristics of the user comprises:
receiving an image of the user's face; and
selecting a three-dimensional model of a face of the stored library of three-dimensional models of faces having at least a threshold similarity to the image of the user's face.

6. The method of claim 1 , wherein the set of animation parameters comprises a vector of blendshape coefficients that determines a weight of different facial expressions in a linear combination.

7. The method of claim 1 , wherein determining one or more transformations mapping images captured by the one or more image capture devices and captured by the one or more additional image capture devices into three dimensions comprises:
determining one or more transformations mapping images captured by the one or more image capture devices and captured by the one or more additional image capture devices into three dimensions; and
determining one or more differential transformations between pairs of image capture devices or pairs of additional image capture devices.

8. The method of claim 7 , wherein determining one or more differential transformations between pairs of image capture devices or pairs of additional image capture devices comprises:
determining six differential transformations between pairs of image capture devices and pairs of additional image capture devices.

9. The method of claim 1 , further comprising:
obtaining images of portions of the user's body outside of the HMD captured by a body tracking system coupled to the HMD;
identifying points corresponding to features of the portions of the user's body within the images of the portions of the user's body;
mapping the identified points to a three-dimensional model of a body;
generating a set of body animation parameters for each portion of the user's body within the images captured by the body tracking system;
determining one or more transformations mapping images captured by the body tracking system into three dimensions;
generating a body rendering model based on the sets of body animation parameters and the determined transformations;
repositioning one or more points of the three-dimensional model of the body by applying the body rendering model to the three-dimensional model of the body; and
generating content presenting the three-dimensional model of the body with the repositioned one or more points.

10. The method of claim 9 , wherein the features of the portions of the user's body comprise one or more joints of the portions of the user's body.

11. A computer program product comprising a non-transitory computer readable storage medium having instructions encoded thereon that, when executed by a processor, cause the processor to:
obtain images of portions of a user's face enclosed by a head mounted device (HMD) captured by one or more image capture devices coupled to the HMD;
obtain images of additional portions of the user's face outside of the HMD captured by one or more additional image capture devices coupled to the HMD;
identify points corresponding to features of the portions of the user's face within the images of the portions of the user's face and within the images of the additional portions of the user's face;
determine regularization parameters for different portions of the user's face based on the images captured by the one or more image capture devices and captured by the one or more additional image capture devices;
map the identified points to a three-dimensional model of a face;
generate a set of animation parameters for each portion of the user's face and for each additional portion of the user's face from identified points within the images captured by the one or more image capture devices and captured by the one or more additional image capture devices;
apply the corresponding determined regularization parameters to the corresponding generated set of animation parameters for each portion of the user's face and for each additional portion of the user's face to generate a modified set of animation parameters for each portion of the user's face and for each additional portion of the user's face;
determine one or more transformations mapping images captured by the one or more image capture devices and captured by the one or more additional im

Assignees

• Facebook Tech Llc

Inventors

• Sheikh Yaser
• Badino Hernan
• Hypes Alexander Trenor
• Wang Dawei
• Shahmohammadi Mohsen
• Perdoch Michal
• Saragih Jason
• Wei Shih-En

Classifications

• G06T13/40 Primary

  of characters, e.g. humans, animals or virtual beings · CPC title

• G06T2200/08

  involving all processing steps from image acquisition to 3D model generation · CPC title

• G06T17/00

  Three-dimensional [3D] modelling for computer graphics · CPC title

• G06K9/00302

  Physics · mapped topic

• G06K9/00281

  Physics · mapped topic