Generating an avatar from real time image data

Patent metadata

Field	Value
Publication number	US-9697635-B2
Application number	US-201615295667-A
Country	US
Kind code	B2
Filing date	Oct 17, 2016
Priority date	Nov 1, 2013
Publication date	Jul 4, 2017
Grant date	Jul 4, 2017

Abstract

Technology is disclosed for automatically generating a facial avatar resembling a user in a defined art style. One or more processors generate a user 3D head model for the user based on captured 3D image data from a communicatively coupled 3D image capture device. A set of user transferable head features from the user 3D head model are automatically represented by the one or more processors in the facial avatar in accordance with rules governing transferable user 3D head features. In some embodiments, a base or reference head model of the avatar is remapped to include the set of user head features. In other embodiments, an avatar head shape model is selected based on the user 3D head model, and the transferable user 3D head features are represented in the avatar head shape model.

First claim

What is claimed is:

1. One or more processor readable storage devices having encoded thereon instructions for causing one or more processors to perform a method for automatically generating a facial avatar resembling a user in a defined art style, the method comprising:
generating a user head model for the user based on captured image data by one or more processors communicatively coupled to a capture device;
identifying a set of transferable user head features from the user head model which can be represented in the facial avatar based on transferable head feature rules for the defined art style;
identifying a set of avatar character head features to represent the set of transferable user head features based on the transferable head feature rules for the defined art style by:
identifying one or more user head features which can be represented by one or more symbol features in the defined art style of the avatar based on transferable head feature rules for the defined art style; and
representing the one or more user head features by the one or more symbol features into a model for a selected avatar head shape;

generating the facial avatar in the defined art style including the set of avatar character head features;
displaying the facial avatar by a display communicatively coupled to the one or more processors;
updating a facial expression of the facial avatar responsive to changes detected in user facial features; and
updating display of the facial avatar with the updated facial expression.

2. The one or more processor readable storage devices of claim 1 wherein generating the user head model for the user based on the captured image data by one or more processors communicatively coupled to the capture device further comprises:
generating by the one or more processors a point cloud based on the captured image data;
identifying the head features including the facial features in the point cloud;
fitting a tracking mesh to the head features including the facial features in the point cloud;
determining a set of measurements relating the head features including the facial features of the 3D mesh by a distance, an angle or a proportion; and
storing the set of measurements and the mesh as the user head model.

3. The one or more processor readable storage devices of claim 2 wherein determining the set of measurements relating the head features including the facial features of the mesh by a distance, an angle or a proportion further comprises:
determining a head size and head shape from the tracking mesh,
selecting a human head reference mesh model with N predefined shape units based on the head size and head shape,
identifying one or more corresponding points of the head features in the tracking mesh and the human head reference mesh model, and
iteratively performing a morph target animation technique operating on the shape units of the human head reference mesh model until a matching criteria with the tracking mesh is satisfied; and
wherein storing the set of measurements and the mesh as the user head model further comprises storing as coefficient data of the user head model the coefficients data for each point in each respective shape unit N which satisfied the matching criteria.

4. The one or more processor readable storage devices of claim 1 wherein generating the facial avatar in the defined art style including the set of avatar character head features representing the set of transferable user head features further comprises:
retrieving a base avatar head mesh of the defined art style;
determining coefficient data identifying a shape difference between the user head model for each shape unit N in the set of transferable user head features and the respective shape unit N in the base avatar head mesh;
remapping the shape units of the base avatar head mesh using a morph target animation technique using the determined coefficient data and a remapping function identified for each shape unit in remapping rules associated with the avatar selection; and
storing the remapped base avatar head mesh as the facial avatar associated with and resembling the user.

5. The one or more processor readable storage devices of claim 4 further comprising:
performing blending of the shape units of the remapped base avatar head mesh for satisfying blending criteria; and
adding texture and color to the remapped base avatar head mesh personalized for the user based on rules associated with head features in the defined art style of the avatar.

6. The one or more processor readable storage devices of claim 1 wherein updating the facial expression of the facial avatar responsive to changes detected in user facial features further comprises:
periodically determining action measurements of a set of facial features based on stored 3D facial expression metrics for tracking facial changes based on periodically captured image data of the user head features; and
responsive to the action measurements indicating a change in a user facial feature satisfying facial action criteria,
identifying shape units of the user head model effected by the facial changes;
determining action unit coefficients for action unit sub-meshes for the effected shape units, and
updating a facial expression of the facial avatar for mimicking the change in the user facial feature in accordance with the action measurements.

7. A method for automatically generating a facial avatar resembling a user in a defined art style comprising:
capturing image data of user head features including facial features by a capture device;
generating a user head model for the user based on the captured image data by one or more processors communicatively coupled to the capture device;
identifying a set of transferable user head features from the user head model which can be represented in the facial avatar based on transferable head feature rules for the defined art style;
identifying a set of avatar character head features to represent the set of transferable user head features based on the transferable head feature rules for the defined art style by:
identifying one or more user head features which can be represented by one or more symbol features in the defined art style of the avatar based on transferable head feature rules for the defined art style,
representing the one or more user head features by the one or more symbol features into a 3D model for a selected avatar head shape, and wherein
the transferable head feature rules directing which of the facial features are transferable to the facial avatar for the defined art style;

generating the facial avatar in the defined art style including the set of avatar character head features; and
displaying the facial avatar by a display communicatively coupled to the one or more processors.

8. The method of claim 7 wherein generating the user head model for the user based on the captured image data by one or more processors communicatively coupled to the capture device further comprises:
generating by the one or more processors a point cloud based on the captured image data;
identifying the head features including the facial features in the point cloud;
fitting a mesh to the head features including the facial features in the point cloud;
determining the set of measurements relating the head features including the facial features of the mesh by at least one of a distance, an angle or a proportion; and
storing the set of measurements and the mesh as the user head model.

9. The method of claim 8 where fitting a mesh to the head features including the facial features in the point cloud further comprises:
generating a tracking mesh for the head features including the facial features in the point cloud.

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Quinn David J
• Ridgway Peter Alan
• Burton Nicholas David
• Clark Carol
• Hill David T
• Huitema Christian F
• Smith Yancey C
• Winchester Royal D
• Mcfadzen Iain A
• Bastable Andrew John

Classifications

• G06T2207/20036

  Morphological image processing · CPC title

• G06T7/0046

  Physics · mapped topic

• G06K9/00315

  Physics · mapped topic

• G06F3/012

  Head tracking input arrangements · CPC title

• G06T2219/2024

  Style variation · CPC title