Systems and methods for executing improved iterative optimization processes to personify blendshape rigs

Patent metadata

Field	Value
Publication number	US-11189084-B2
Application number	US-202016746423-A
Country	US
Kind code	B2
Filing date	Jan 17, 2020
Priority date	Jul 29, 2016
Publication date	Nov 30, 2021
Grant date	Nov 30, 2021

Abstract

The present specification describes systems and methods for automatically generating personalized blendshapes from actor performance measurements, while preserving the semantics of a template facial animation rig. The disclosed inventions facilitate the creation of an ensemble of realistic digital double face rigs for each individual with consistent behaviour across the set with sophisticated iterative optimization techniques.

First claim

We claim:

1. A computer-implemented method for generating and dynamically modifying a blendshape within a graphical user interface rendered in a display, wherein the method is implemented in a computer having a processor, and wherein the processor is in data communication with the display and with a storage unit, the method comprising:
acquiring from the storage unit a plurality of template blendshapes, wherein each template blendshape is defined by data representative of a plurality of vertices and relationships between said vertices that, when rendered onto said display, visually represent at least one facial expression;
acquiring a plurality of facial expression measurements, wherein each facial expression measurement is defined by data representative of at least one facial expression captured from a physical performance by an actor;
using said computer and at least a portion of said plurality of facial expression measurements to generate an initial blendshape; and
executing an iterative optimization process, within said computer, to generate an output blendshape, wherein said iterative optimization process applies a plurality of transformations to the initial blendshape based upon a first variable representative of a degree of sparseness, a second variable representative of a degree of temporal smoothness, a third variable representative of a degree of deformation regularization, and a fourth variable representative of a degree of direction of motion regularization and wherein said iterative optimization process iteratively adjusts each of said first, second, third, and fourth variable to generate the output blendshape.

2. The computer-implemented method of claim 1 further comprising displaying an icon on said display, wherein said icon is adapted to be manipulated and wherein, upon a manipulation, the first variable is modified, thereby causing the degree of sparseness to increase or decrease.

3. The computer-implemented method of claim 1 further comprising displaying an icon on said display, wherein said icon is adapted to be manipulated and wherein, upon a manipulation, the second variable is modified, thereby causing the degree of temporal smoothness to increase or decrease.

4. The computer-implemented method of claim 1 further comprising displaying an icon on said display, wherein said icon is adapted to be manipulated and wherein, upon a manipulation, the third variable is modified, thereby causing the degree of deformation regularization to increase or decrease.

5. The computer-implemented method of claim 1 further comprising displaying an icon on said display, wherein said icon is adapted to be manipulated and wherein, upon a manipulation, the fourth variable is modified, thereby causing the degree of direction of motion regularization to increase or decrease.

6. The computer-implemented method of claim 1 wherein at least one of said plurality of transformations factors out rigid motion when computing a plurality of weights.

7. The computer-implemented method of claim 1 wherein the output blendshape, x i , is defined by

min

w
i

,

R
i

,

t
i

,
D
,

b
0

⁢

∑

i
=
1

n
f

⁢

⁢

E
g
i

,

(
2
)

where
E i g =∥M i ( X i −p i )∥ 2 ,

X i =( I nv ⊗R i )( Dw i +b 0 )+(1 nv ⊗t i ).
wherein the initial blendshape is defined by a pose offset D, a neutral pose b 0 , and blendshape weights w i , wherein rotation R i and translation t i represent rigid motion at an i th frame, wherein p i is data representing at least a portion of said plurality of facial expression measurements, wherein M i is a square diagonal matrix where each diagonal element stores a matching confidence value of each vertex, wherein I nv is an identity matrix with a size equal to a number of vertices n v , and wherein 1 nv is a column vector of ones with a length of n v .

8. The computer-implemented method of claim 1 wherein the output blendshape relative to a target facial expression has an average fitting error of less than 1.8 mm.

9. The computer-implemented method of claim 1 wherein the target facial expression is at least one of a smile, a laugh, a frown, a growl, a yell, closed eyes, open eyes, heightened eyebrows, lowered eyebrows, pursed lips, a mouth shape of a vowel, or a mouth shape of a consonant.

10. The computer-implemented method of claim 1 further comprising outputting into a random access memory at least one of a plurality of weights or a plurality of rigid motions associated with said output blendshape.

11. A system for generating and dynamically modifying a blendshape within a graphical user interface rendered in a display, wherein the system comprises a computer having a processor in data communication with the display and a storage unit storing a plurality of programmatic instructions, and wherein the plurality of programmatic instructions, when executed by said processor, cause the system to:
acquire from the storage unit a plurality of template blendshapes, wherein each template blendshape is defined by data representative of a plurality of vertices and relationships between said vertices that, when rendered onto said display, visually represent at least one facial expression;
acquire a plurality of facial expression measurements, wherein each facial expression measurement is defined by data representative of at least one facial expression captured from a physical performance by an actor;
use at least a portion of said plurality of facial expression measurements to generate an initial blendshape; and
execute an iterative optimization process to generate an output blendshape, wherein said iterative optimization process applies to the initial blendshape a first transformation indicative of a degree of sparseness, a second transformation indicative of a degree of temporal smoothness, a third transformation indicative of a degree of deformation regularization, and a fourth transforma

Assignees

• Activision Publishing Inc

Inventors

• Ma Iii Wan-Chun
• Ma Chongyang

Classifications

• G06V40/174

  Facial expression recognition · CPC title

• G06T15/503 Primary

  Blending, e.g. for anti-aliasing · CPC title

• G06T13/40

  of characters, e.g. humans, animals or virtual beings · CPC title

• G06K9/00302

  Physics · mapped topic