Generating a facial-hair-free mesh of a subject

Patent metadata

Field	Value
Publication number	US-12367649-B2
Application number	US-202318102480-A
Country	US
Kind code	B2
Filing date	Jan 27, 2023
Priority date	Jan 27, 2022
Publication date	Jul 22, 2025
Grant date	Jul 22, 2025

Abstract

Methods and systems for generating three-dimensional (3D) models and facial hair models representative of subjects (e.g., actors or actresses) using facial scanning technology. Initial subject facial data, including facial frames and facial performance frames (e.g., images of the subject collected from a capture system) can be used to accurately predict the structure of the subject's face underneath their facial hair to produce a reference 3D facial shape of the subject. Likewise, image processing techniques can be used to identify facial hairs and generate a reference facial hair model. The reference 3D facial shape and reference facial hair mode can subsequently be used to generate performance 3D facial shapes and a performance facial hair model corresponding to a performance by the subject (e.g., reciting dialog).

First claim

What is claimed is:

1. A computer-implemented method of generating a reference three-dimensional (3D) facial shape corresponding to a subject, the method comprising performing, by a computer system:
retrieving initial subject facial data comprising an initial reference 3D facial shape, wherein the initial reference 3D facial shape represents the subject and comprising a plurality of initial reference geometric elements;
determining a facial hair mask based on the initial subject facial data, the facial hair mask defining a plurality of probabilities corresponding to the plurality of initial reference geometric elements, wherein each probability of the plurality of probabilities indicates a probability that a corresponding initial geometric element represents facial hair;
determining a plurality of reference facial shape weights corresponding to the plurality of initial reference geometric elements using the facial hair mask;
generating or retrieving an estimate 3D facial shape; and
generating the reference 3D facial shape by combining the estimate 3D facial shape and the initial reference 3D facial shape, wherein the reference 3D facial shape comprises a plurality of reference geometric elements, wherein the reference 3D facial shape represents the subject without facial hair.

2. The computer-implemented method of claim 1 , wherein the estimate 3D facial shape comprises a component estimate 3D facial shape, and wherein generating or retrieving the estimate 3D facial shape comprises:
retrieving or generating a mean component 3D facial shape and a plurality of component 3D facial shapes; and
generating, using an optimization solver, the component estimate 3D facial shape comprising a plurality of component estimate geometric elements, the component estimate 3D facial shape comprising a weighted combination of the mean component 3D facial shape and the plurality of component 3D facial shapes, wherein the optimization solver is constrained by a reference facial shape error function relating the plurality of component estimate geometric elements, the plurality of initial reference geometric elements, and the plurality of reference facial shape weights.

3. The computer-implemented method of claim 2 , wherein the plurality of component 3D facial shapes comprise a plurality of principal component 3D facial shapes, and wherein retrieving or generating the mean component 3D facial shape and the plurality of component 3D facial shapes comprises:
retrieving a plurality of facial hair free 3D facial shapes corresponding to a plurality of subjects without facial hair;
determining the mean component 3D facial shape based on the plurality of facial hair free 3D facial shapes; and
determining the plurality of principal component 3D facial shapes by performing a principal component analysis on the plurality of facial hair free 3D facial shapes, wherein the plurality of principal component 3D facial shapes comprise a plurality of normalized eigenvectors produced using the principal component analysis.

4. The computer-implemented method of claim 2 , wherein:
the reference facial shape error function comprises a weighted combination of a positional energy term, an iterative closest points energy term, and an L2 regularizer; and
the positional energy term is proportional to a weighted difference between the plurality of component estimate geometric elements and the plurality of initial reference geometric elements, wherein the weighted difference is weighted using the plurality of reference facial shape weights.

5. The computer-implemented method of claim 2 , wherein generating the component estimate 3D facial shape using the optimization solver comprises:
determining, using the optimization solver, a plurality of component 3D facial shape weights corresponding to the plurality of component 3D facial shapes;
weighing each component 3D facial shape with a corresponding component 3D facial shape weight, thereby producing a plurality of weighted component 3D facial shapes; and
generating the component estimate 3D facial shape by performing a linear combination of the weighted component 3D facial shapes and the mean component 3D facial shape.

6. The computer-implemented method of claim 1 , wherein the estimate 3D facial shape comprises a captured 3D facial shape corresponding to the subject without facial hair.

7. The computer-implemented method of claim 1 , wherein the facial hair mask comprises a UV texture map generated using a plurality of raw reconstruction errors corresponding to the initial reference 3D facial shape.

8. The computer-implemented method of claim 1 , wherein determining the plurality of reference facial shape weights corresponding to the plurality of initial reference geometric elements using the facial hair mask comprises:
determining one or more facial hair probability thresholds; and
for each initial reference geometric element, comparing a corresponding probability from the facial hair mask to the one or more facial hair probability thresholds and determining a corresponding reference facial shape weight based on this comparison, thereby determining the plurality of reference facial shape weights.

9. The computer-implemented method of claim 1 , further comprising, prior to generating the reference 3D facial shape by combining the estimate 3D facial shape and the initial reference 3D facial shape:
performing a refinement mesh deformation process on the estimate 3D facial shape using an optimization solver, wherein the optimization solver is constrained by a refinement mesh deformation error function comprising a weighted sum of a positional energy term and a Laplacian regularizer term.

10. The computer-implemented method of claim 1 , wherein the initial subject facial data includes a reference facial frame comprising one or more facial images of the subject each comprising a plurality of reference pixels, and wherein the computer-implemented method further comprises:
retrieving a plurality of facial performance frames corresponding to a facial performance by the subject, wherein each facial performance frame comprises one or more facial performance images of the subject each comprising a plurality of performance pixels;
determining a set of facial hair free reference pixels from the reference facial frame using the facial hair mask;
for each facial performance frame of the plurality of facial performance frames, determining a set of facial hair free performance pixels using the facial hair mask, thereby determining a plurality of sets of facial hair free performance pixels;
performing a pixel motion estimate process between the sets of facial hair free reference pixels and each set of facial hair free performance pixels, thereby determining a plurality of pixel motion estimates corresponding to the plurality of facial performance frames;
determining a plurality of facial shape transformations corresponding to the plurality of facial performance frames, wherein each facial shape transformation comprises a facial hair free transformation component and a facial hair transformation component, wherein the facial hair free transformation component is derived from a corresponding pixel motion estimate of the plurality of pixel motion estimates, wherein the facial hair transformation component comprises a semi-rigid transformation based on the facial hair free transformation component; and
generating a plurality of performance 3D facial shapes by applying the plurality of facial shape transformation to the reference 3D facial shape, each performance 3D facial shape of the plurality of performance 3D facial shapes corresponding to a facial performance frame of the plurality of facial performance frames, eac

Assignees

• Disney Entpr Inc
• Eth Zuerich Eidgenoessische Technische Hochschule Zuerich

Inventors

• Winberg Sebastian
• Chandran Prashanth
• Gotardo Paulo Fabiano Urnau
• Zoss Gaspard
• Bradley Derek Edward

Classifications

• G06T17/20

  Finite element generation, e.g. wire-frame surface description, {tesselation} · CPC title

• G06T13/40 Primary

  of characters, e.g. humans, animals or virtual beings · CPC title

• G06T2207/10016

  Video; Image sequence · CPC title

• G06T2207/20076

  Probabilistic image processing · CPC title

• G06T2207/30201

  Face · CPC title