Dynamic facial hair capture of a subject

Patent metadata

Field	Value
Publication number	US-12361663-B2
Application number	US-202318102498-A
Country	US
Kind code	B2
Filing date	Jan 27, 2023
Priority date	Jan 27, 2022
Publication date	Jul 15, 2025
Grant date	Jul 15, 2025

Abstract

Embodiments of the present disclosure are directed to methods and systems for generating three-dimensional (3D) models and facial hair models representative of subjects (e.g., actors or actresses) using facial scanning technology. Methods accord to embodiments may be useful for performing facial capture on subjects with dense facial hair. Initial subject facial data, including facial frames and facial performance frames (e.g., images of the subject collected from a capture system) can be used to accurately predict the structure of the subject's face underneath their facial hair to produce a reference 3D facial shape of the subject. Likewise, image processing techniques can be used to identify facial hairs and generate a reference facial hair model. The reference 3D facial shape and reference facial hair mode can subsequently be used to generate performance 3D facial shapes and a performance facial hair model corresponding to a performance by the subject (e.g., reciting dialog).

First claim

What is claimed is:

1. A computer-implemented method of generating a reference facial hair model that represents facial hair of a subject, the computer-implemented method comprising performing, by a computer system:
retrieving initial subject facial data comprising a plurality of facial frames of the subject, each facial frame comprising one or more facial images of the subject, wherein the plurality of facial frames comprise a reference facial frame and a plurality of non-reference facial frames;
for each facial frame of the plurality of facial frames, performing a facial hair identification process, thereby determining a plurality of initial reference facial hair data elements and a plurality of sets of non-reference facial hair data elements, wherein the plurality of initial reference facial hair data elements and the plurality of sets of non-reference facial hair data elements represent facial hair of the subject;
for each set of non-reference facial hair data elements, determining a set of projected non-reference facial hair data elements, thereby determining a plurality of sets of projected non-reference facial hair data elements;
generating using an optimization solver, for each set of projected non-reference facial hair data elements, a set of alignment transformations, thereby determining a plurality of sets of alignment transformations, wherein the optimization solver is constrained by a facial hair alignment error function relating the set of alignment transformations to the set of projected non-reference facial hair data elements;
applying the plurality of sets of alignment transformations to the plurality of sets of non-reference facial hair data elements, thereby determining a plurality of sets of aligned non-reference facial hair data elements;
combining the plurality of sets of aligned non-reference facial hair data elements and the plurality of initial reference facial hair data elements, thereby determining a plurality of reference facial hair data elements, wherein the reference facial hair model comprises the plurality of reference facial hair data elements, wherein the plurality of reference facial hair data elements comprises a plurality of facial hair sets of reference facial hair data elements, each facial hair set comprising a three-dimensional representation of a different facial hair of the subject;
retrieving a plurality of facial performance frames corresponding to a facial performance by the subject, wherein each facial performance frame comprises one or more facial images of the subject;
for each facial performance frame of the plurality of facial performance frames, performing an optical flow projection process on the reference facial hair model, thereby determining a set of projected reference facial hair data elements, thereby determining a plurality of sets of performance facial hair data elements corresponding to the plurality of facial frames;
for each facial performance frame of the plurality of facial performance frames, determining a set of projected reference facial hair data elements, thereby determining a plurality of sets of projected reference facial hair data elements corresponding to the plurality of facial performance frames;
for each facial performance frame of the plurality of facial performance frames, generating using an optimization solver, a set of reference alignment transformations, wherein the optimization solver is constrained by a facial hair performance error function relating the set of reference alignment transformations to a corresponding set of projected reference facial hair data elements, thereby determining a plurality of sets of reference alignment transformations; and
applying the plurality of sets of reference alignment transformations to the plurality of reference facial hair data elements, thereby determining a plurality of sets of aligned reference facial hair data elements, wherein a performance facial hair model comprises a plurality of sets of performance facial hair data elements comprising the plurality of sets of aligned reference facial hair data elements;
retrieving or generating a reference 3D facial shape, wherein the reference 3D facial shape represents a face of the subject, wherein the reference 3D facial shape comprises a plurality of reference geometric elements;
retrieving or determining a facial hair mask comprising a plurality of probabilities corresponding to a plurality of facial regions on a face of the subject, each probability indicating the probability that facial hair is located within a corresponding facial region;
retrieving or generating a plurality of performance 3D facial shapes that represent the face of the subject during the facial performance, each performance 3D facial shape comprising a plurality of performance geometric elements;
determining a set of reference regions corresponding to the reference 3D facial shape, each reference region of the set of reference regions corresponding to a plurality of reference region geometric elements from the plurality of reference geometric elements;
determining a plurality of sets of performance regions corresponding to the plurality of performance 3D facial shapes, each performance region of the plurality of sets of performance regions corresponding to a plurality of performance region geometric elements from the plurality of performance geometric elements;
generating a plurality of performance refinement weights using the facial hair mask;
determining, based on the set of reference regions, a set of subsets of reference facial hair data elements from the plurality of reference facial hair data elements from the reference facial hair model, wherein each subset of reference facial hair data elements corresponds to a reference region of the set of reference regions;
determining, based on the plurality of sets of performance regions, a plurality of sets of subsets of performance facial hair data elements from the plurality of sets of performance facial hair data elements from the performance facial hair model, wherein each subset of performance facial hair data elements corresponds to a performance region of the plurality of sets of performance regions;
for each subset of performance facial hair data elements of the plurality of sets of subsets of performance facial hair data elements, determining a facial hair transformation between that subset of performance facial hair data elements and a corresponding subset of reference facial hair data elements, thereby determining a plurality of facial hair transformations;
applying the plurality of facial hair transformations to the plurality of reference regional geometric elements, thereby determining a plurality of transformed reference regional geometric elements; and
for each facial performance frame of the plurality of facial performance frames, generating a refined performance 3D facial shape using an optimization solver, wherein the optimization solver is constrained based on a refined performance error function relating the plurality of transformed reference regional geometric elements, the plurality of performance geometric elements corresponding to that facial frame, and the plurality of performance refinement weights, the computer system thereby generating a plurality of refined performance 3D facial shapes.

2. The computer-implemented method of claim 1 , wherein the facial hair alignment error function comprises a weighted combination of an iterative closest points energy term, an optical flow energy term, and a neighborhood regularizer.

3. The computer-implemented method of claim 1 , wherein the plurality of sets of alignment transformations are defined by a plurality of translation vectors and quaternion rotations.

4. The computer-implemented method of claim 1 , further comprising:
determining a plurality of facial hair sets of r

Assignees

• Disney Entpr Inc
• Eth Zuerich Eidgenoessische Technische Hochschule Zuerich

Inventors

• Bradley Derek Edward
• Gotardo Paulo Fabiano Urnau
• Zoss Gaspard
• Chandran Prashanth
• Winberg Sebastian

Classifications

• G06T17/20

  Finite element generation, e.g. wire-frame surface description, {tesselation} · CPC title

• G06T13/40 Primary

  of characters, e.g. humans, animals or virtual beings · CPC title

• G06T2207/10016

  Video; Image sequence · CPC title

• G06T2207/20076

  Probabilistic image processing · CPC title

• G06T2207/30201

  Face · CPC title