Content retargeting using facial layers

Patent metadata

Field	Value
Publication number	US-9245176-B2
Application number	US-201213564476-A
Country	US
Kind code	B2
Filing date	Aug 1, 2012
Priority date	Aug 1, 2012
Publication date	Jan 26, 2016
Grant date	Jan 26, 2016

Abstract

Techniques are disclosed for retargeting facial expressions. Input is received that represents a facial expression of a first character. Facial layers are generated based on the received input. The facial layers include one or more parameters extracted from the received input. A facial expression for a second character and corresponding to the facial expression of the first character is generated, based on the facial layers and without defining any spatial correspondence between the first character and the second character.

First claim

What is claimed is:

1. A computer-implemented method to retarget facial expressions by providing a parameter-parallel retargeting space between an input facial expression and an output facial expression and via a plurality of facial layers, the computer-implemented method comprising:
receiving input representing a facial expression of a first character;
generating a plurality of facial layers for the first character, based on the received input and a composition function and by operation of one or more computer processors, wherein the plurality of facial layers includes a plurality of parameters extracted from the received input, the plurality of parameters including a simplex coefficient common to the plurality of facial layers and weighted by a respective measure of influence for each of the plurality of facial layers; wherein the simplex coefficient is extracted by an optimization operation that includes determining a set of inputs for which a given function attains a minimum value, wherein the optimization operation is subject to a set of constraints and is based on a sequential quadratic programming algorithm; wherein each facial layer encodes, in a simplicial basis, one or more semantically significant aspects of the facial expression of the first character; and wherein a simplex is formed from components of each simplicial basis; and
generating, for a second character different from the first character in appearance, a facial expression corresponding to the facial expression of the first character, based on the plurality of facial layers.

2. The computer-implemented method of claim 1 , wherein the facial expression for the second character includes a plurality of facial features of the second character, wherein each of the plurality of facial features is modeled independently based on the plurality of facial layers.

3. The computer-implemented method of claim 2 , wherein the plurality of parameters spans a space of facial expression.

4. The computer-implemented method of claim 3 , wherein the space of facial expression comprises a predetermined association of each of a plurality of different facial expressions with a distinct emotion corresponding to the respective facial expression.

5. The computer-implemented method of claim 4 , wherein the parameter-parallel retargeting space is induced, wherein generating the facial expression for the second character comprises:
generating a plurality of facial layers for the second character;
transferring the plurality of parameters from the plurality of facial layers for the first character to the plurality of facial layers for the second character; and
combining the plurality of facial layers for the second character based on the composition function.

6. A computer program product to retarget facial expressions by providing a parameter-parallel retargeting space between an input facial expression and an output facial expression and via a plurality of facial layers, the computer program product comprising:
a non-transitory computer-readable medium having computer-readable program code embodied therewith, the computer-readable program code executable by one or more computer processors to:
receive input representing a facial expression of a first character;
generate a plurality of facial layers for the first character, based on the received input and a composition function, wherein the plurality of facial layers includes a plurality of parameters extracted from the received input, the plurality of parameters including a simplex coefficient common to the plurality of facial layers and weighted by a respective measure of influence for each of the plurality of facial layers; wherein the simplex coefficient is extracted by an optimization operation that includes determining a set of inputs for which a given function attains a minimum value, wherein the optimization operation is subject to a set of constraints and is based on a sequential quadratic programming algorithm; wherein each facial layer encodes, in a simplicial basis, one or more semantically significant aspects of the facial expression of the first character; and wherein a simplex is formed from components of each simplicial basis; and

generate, for a second character different from the first character in appearance, a facial expression corresponding to the facial expression of the first character, based on the plurality of facial layers.

7. The computer program product of claim 6 , wherein the facial expression for the second character includes a plurality of facial features of the second character, wherein each of the plurality of facial features is modeled independently based on the plurality of facial layers.

8. The computer program product of claim 7 , wherein the plurality of parameters spans a space of facial expression.

9. The computer program product of claim 8 , wherein the space of facial expression comprises a predetermined association of each of a plurality of different facial expressions with a distinct emotion corresponding to the respective facial expression.

10. The computer program product of claim 9 , wherein the parameter-parallel retargeting space is induced, wherein the computer-readable program code is further executable to:
generate a plurality of facial layers for the second character;
transfer the plurality of parameters from the plurality of facial layers for the first character to the plurality of facial layers for the second character; and
combine the plurality of facial layers for the second character based on the composition function.

11. A system to retarget facial expressions by providing a parameter-parallel retargeting space between an input facial expression and an output facial expression and via a plurality of facial layers, the system comprising:
one or more computer processors;
a memory containing a program which, when executed by the one or more computer processors, is configured to perform an operation comprising:
receiving input representing a facial expression of a first character;
generating a plurality of facial layers for the first character, based on the received input and a composition function, wherein the plurality of facial layers includes a plurality of parameters extracted from the received input, the plurality of parameters including a simplex coefficient common to the plurality of facial layers and weighted by a respective measure of influence for each of the plurality of facial layers; wherein the simplex coefficient is extracted by an optimization operation that includes determining a set of inputs for which a given function attains a minimum value, wherein the optimization operation is subject to a set of constraints and is based on a sequential quadratic programming algorithm; wherein each facial layer encodes, in a simplicial basis, one or more semantically significant aspects of the facial expression of the first character; and wherein a simplex is formed from components of each simplicial basis; and
generating, for a second character different from the first character in appearance, a facial expression corresponding to the facial expression of the first character, based on the plurality of facial layers.

12. The system of claim 11 , wherein the facial expression for the second character includes a plurality of facial features of the second character, wherein each of the plurality of facial features is modeled independently based on the plurality of facial layers.

13. The system of claim 12 , wherein the plurality of parameters spans a space of facial expression.

14. The system of claim 13 , wherein the space of facial expression comprises a predetermined associat

Assignees

• Matthews Iain
• Kholgade Natasha
• Sheikh Yaser
• Disney Entpr Inc

Inventors

• Matthews Iain
• Kholgade Natasha
• Sheikh Yaser

Classifications

• G06K9/00315 Primary

  Physics · mapped topic

• G06T17/20

  Finite element generation, e.g. wire-frame surface description, {tesselation} · CPC title

• G06T13/40 Primary

  of characters, e.g. humans, animals or virtual beings · CPC title

• G06V40/176 Primary

  Dynamic expression · CPC title