Generating an avatar from real time image data

Patent metadata

Field	Value
Publication number	US-9508197-B2
Application number	US-201314070481-A
Country	US
Kind code	B2
Filing date	Nov 1, 2013
Priority date	Nov 1, 2013
Publication date	Nov 29, 2016
Grant date	Nov 29, 2016

Abstract

Technology is disclosed for automatically generating a facial avatar resembling a user in a defined art style. One or more processors generate a user 3D head model for the user based on captured 3D image data from a communicatively coupled 3D image capture device. A set of user transferable head features from the user 3D head model are automatically represented by the one or more processors in the facial avatar in accordance with rules governing transferable user 3D head features. In some embodiments, a base or reference head model of the avatar is remapped to include the set of user head features. In other embodiments, an avatar head shape model is selected based on the user 3D head model, and the transferable user 3D head features are represented in the avatar head shape model.

First claim

What is claimed is:

1. A method for automatically generating a facial avatar resembling a user in a defined art style comprising:
capturing three dimensional (3D) image data of user head features including facial features by a 3D capture device;
generating a user 3D head model for the user based on the captured 3D image data by one or more processors communicatively coupled to the 3D capture device;
identifying a set of transferable user head features from the user 3D head model which can be represented in the facial avatar based on transferable head feature rules for the defined art style;
identifying a set of avatar character head features to represent the set of transferable user head features based on the transferable head feature rules for the defined art style, the transferable head feature rules directing which of the facial features are transferable to the facial avatar for the defined art style;
iteratively performing a morph target animation technique operating on shape units of a human head reference mesh model until a matching criteria with a 3D tracking mesh is satisfied,
storing a set of measurements relating the head features and the 3D tracking mesh as a user 3D head model, including storing as coefficient data of the user 3D head model 3D the coefficients data for each point in each respective shape unit which satisfied the matching criteria;
generating the facial avatar in the defined art style including the set of avatar character head features; and
displaying the facial avatar by a display communicatively coupled to the one or more processors.

2. The method of claim 1 wherein generating a user 3D head model for the user based on the captured 3D image data by one or more processors communicatively coupled to the 3D capture device further comprises:
generating by the one or more processors a point cloud based on the captured 3D image data;
identifying head features including facial features in the point cloud;
fitting a 3D mesh to the head features including the facial features in the point cloud;
determining the set of measurements relating the head features including the facial features of the 3D mesh by at least one of a distance, an angle or a proportion
storing.

3. The method of claim 2 where fitting a 3D mesh to the head features including the facial features in the point cloud further comprises:
generating a 3D tracking mesh for the head features including the facial features in the point cloud.

4. The method of claim 3 wherein
determining the set of measurements relating the head features including the facial features of the 3D mesh by at least one of a distance, an angle or a proportion further comprises:
determining a head size and head shape from the 3D tracking mesh,
selecting a human head reference mesh model with N predefined 3D shape units based on the head size and head shape,
identifying one or more corresponding points of head features in the 3D tracking mesh and the human head reference mesh model.

5. The method of claim 4 wherein identifying a set of avatar character head features to represent the set of transferable user head features based on the transferable head feature rules for the defined art style further comprising:
identifying one or more user head features which can be represented by one or more symbol features in the defined art style of the avatar based on transferable head feature rules for the defined art style; and
representing the one or more user head features by the one or more symbol features into the 3D model for the selected avatar head shape.

6. The method of claim 4 wherein representing the set of transferable user head features in a 3D model for the selected avatar head shape further comprises:
representing the selected avatar head shape as point cloud data;
representing the set of transferable user head features and the avatar specific head features outside the set of transferable user head features as point cloud data; and
generating a mesh connecting the point cloud data as the 3D model for the selected avatar head shape.

7. The method of claim 1 wherein generating a facial avatar in the defined art style including the set of avatar character head features further comprises
retrieving measurements relating head features for the set of transferable user head features from the set of measurements relating the head features including the facial features stored for the user 3D head model,
based on a head shape of the user 3D head model, selecting an avatar head shape from available head shapes for the avatar,
representing the set of transferable user head features in a 3D model for the selected avatar head shape,
enhancing the set of transferable user head features in the 3D model with stylistic elements associated with the avatar,
adding avatar specific head features outside the set of transferable user head features,
automatically blending the avatar specific head features and the stylistic elements into the 3D model for the selected avatar head shape, and
storing the 3D model as the facial avatar associated with and resembling the user.

8. The method of claim 1 wherein updating a facial expression of the facial avatar responsive to changes detected in user facial features further comprises:
periodically determining 3D action measurements of a set of facial features based on stored 3D facial expression metrics for tracking facial changes based on periodically captured 3D image data of user head features; and
responsive to the 3D action measurements indicating a change in a user facial feature satisfying facial action criteria, updating a facial expression of the facial avatar to mimic the change in the user facial feature in accordance with the 3D action measurements.

9. One or more processor readable storage devices having encoded thereon instructions for causing one or more processors to perform a method for automatically generating a facial avatar resembling a user in a defined art style, the method comprising:
generating a user 3D head model for the user based on captured 3D image data by one or more processors communicatively coupled to a 3D capture device;
identifying a set of transferable user head features from the user 3D head model which can be represented in the facial avatar based on transferable head feature rules for the defined art style;
identifying a set of avatar character head features to represent the set of transferable user head features based on the transferable head feature rules for the defined art style by:
identifying one or more user head features which can be represented by one or more symbol features in the defined art style of the avatar based on transferable head feature rules for the defined art style; and
representing the one or more user head features by the one or more symbol features into a 3D model for a selected avatar head shape;

generating the facial avatar in the defined art style including the set of avatar character head features;
displaying the facial avatar by a display communicatively coupled to the one or more processors;
updating a facial expression of the facial avatar responsive to changes detected in user facial features; and
updating display of the facial avatar with the updated facial expression.

10. The one or more processor readable storage devices of claim 9 wherein generating a user 3D head model for the user based on the captured 3D image data by one or more processors communicatively coupled to the 3D capture device further comprises:
generating by the one or more processors a point cloud based on the captured 3D image data;
identifying head features including facial features in the point cloud;
fitting a 3D tracking mesh to

Assignees

• Microsoft Corp
• Microsoft Technology Licensing Llc

Inventors

• Quinn David J
• Ridgway Peter Alan
• Burton Nicholas David
• Clark Carol
• Hill David T
• Huitema Christian F
• Smith Yancey C
• Winchester Royal D
• Mcfadzen Iain A
• Bastable Andrew John

Classifications

• G06T19/20 Primary

  Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts · CPC title

• G06K9/00315

  Physics · mapped topic

• A63F13/213

  comprising photodetecting means, e.g. cameras, photodiodes or infrared cells (A63F13/219 takes precedence) · CPC title

• G06T17/20

  Finite element generation, e.g. wire-frame surface description, {tesselation} · CPC title

• A63F13/655

  by importing photos, e.g. of the player · CPC title