Image regularization and retargeting system

Patent metadata

Field	Value
Publication number	US-11908057-B2
Application number	US-202117518521-A
Country	US
Kind code	B2
Filing date	Nov 3, 2021
Priority date	Sep 7, 2015
Publication date	Feb 20, 2024
Grant date	Feb 20, 2024

Abstract

Systems and methods for image retargeting are provided. Image data may be acquired that includes motion capture data indicative of motion of a plurality of markers disposed on a surface of a first subject. Each of the markers may be associated with a respective location on the first subject. A plurality of blendshapes may be calculated for the motion capture data based on a configuration of the markers. An error function may be identified for the plurality of blendshapes, and it may be determined that the plurality of blendshapes can be used to retarget a second subject based on the error function. The plurality of blendshapes may then be applied to a second subject to generate a new animation.

First claim

The invention claimed is:

1. A computer-implemented method for retargeting from an image or sequence of images, the method comprising:
identifying a body surface portrayed in a reference image of a physical subject based on respective body surface marker points in the reference image;
generating a representation of the body surface by reference to a group of known body surface positions that correspond with one or more facial muscles associated with a predetermined facial expression, the representation generated by applying a weight to at least one of the known body surface positions and further by utilization of L1-norm regularization via a physiological based energy equation;
retargeting of the representation of the body surface determined according to selection of a plurality of shape primitives; and
based on the retargeting, performing the predetermined facial expression due to a result of evaluating the plurality of shape primitives according to an L1-norm energy minimization function reaching a required minimum activation energy threshold of the predetermined facial expression.

2. The computer-implemented method of claim 1 , wherein one or more of the respective facial muscles associated with the known body surface positions comprise an active muscle state.

3. The computer-implemented method of claim 2 , wherein determining selection of a plurality of shape primitives comprises:
identifying a previous shape weight associated with a respective facial muscle referenced in the representation of the body surface, the previous shape weight indicating a contraction of the respective facial muscle.

4. The computer-implemented method of claim 3 , further comprising:
applying a temporal response function with the identified previous shape weight for the respective facial muscle during the evaluating of the plurality of shape primitives according to an energy minimization function.

5. The computer-implemented method of claim 4 , wherein applying the temporal response function comprises:
adjusting the temporal response function to account for a contracted respective facial muscle.

6. The computer-implemented method of claim 5 , wherein the contracted respective facial muscle represents a facial muscle contraction prior to initiation of performing of the predetermined facial expression.

7. A non-transitory computer readable medium having stored thereon software instructions that, when executed by a processor, cause the processor to perform a series of operations comprising:
identifying a body surface portrayed in a reference image of a physical subject based on respective body surface marker points in the reference image;
generating a representation of the body surface by reference to a group of known body surface positions that correspond with one or more facial muscles associated with a predetermined facial expression, the representation generated by applying a weight to at least one of the known body surface positions and further by utilization of L1-norm regularization via a physiological based energy equation;
retargeting of the representation of the body surface determined according to selection of a plurality of shape primitives; and
based on the retargeting, performing the predetermined facial expression due to a result of evaluating the plurality of shape primitives according to an energy minimization function reaching a required minimum activation energy threshold of the predetermined facial expression.

8. The non-transitory computer readable medium of claim 7 , wherein one or more of the respective facial muscles associated with the known body surface positions comprise an active muscle state.

9. The non-transitory computer readable medium of claim 8 , wherein determining selection of a plurality of shape primitives comprises:
identifying a previous shape weight associated with a respective facial muscle referenced in the representation of the body surface, the previous shape weight indicating a contraction of the respective facial muscle.

10. The non-transitory computer readable medium of claim 9 , further comprising:
applying a temporal response function with the identified previous shape weight for the respective facial muscle during the evaluating of the plurality of shape primitives according to an energy minimization function.

11. The non-transitory computer readable medium of claim 10 , wherein applying the temporal response function comprises:
adjusting the temporal response function to account for a contracted respective facial muscle.

12. The non-transitory computer readable medium of claim 11 , wherein the contracted respective facial muscle represents a facial muscle contraction prior to initiation of performing of the predetermined facial expression.

13. A computer program product comprising a non-transitory computer-readable medium having a computer-readable program code embodied therein to be executed by one or more processors, the program code including instructions to:
identifying a body surface portrayed in a reference image of a physical subject based on respective body surface marker points in the reference image;
generating a representation of the body surface by reference to a group of known body surface positions that correspond with one or more facial muscles associated with a predetermined facial expression, the representation generated by applying a weight to at least one of the known body surface positions and further by utilization of L1-norm regularization via a physiological based energy equation;
retargeting of the representation of the body surface determined according to selection of a plurality of shape primitives; and
based on the retargeting, performing the predetermined facial expression due to a result of evaluating the plurality of shape primitives according to an energy minimization function reaching a required minimum activation energy threshold of the predetermined facial expression.

14. The computer program product of claim 13 , wherein one or more of the respective facial muscles associated with the known body surface positions comprise an active muscle state.

15. The computer program product of claim 14 , wherein determining selection of a plurality of shape primitives comprises:
identifying a previous shape weight associated with a respective facial muscle referenced in the representation of the body surface, the previous shape weight indicating a contraction of the respective facial muscle.

16. The computer program product of claim 15 , further comprising:
applying a temporal response function with the identified previous shape weight for the respective facial muscle during the evaluating of the plurality of shape primitives according to an energy minimization function.

17. The computer program product of claim 16 , wherein applying the temporal response function comprises:
adjusting the temporal response function to account for a contracted respective facial muscle.

18. The computer program product of claim 17 , wherein the contracted respective facial muscle represents a facial muscle contraction prior to initiation of performing of the predetermined facial expression.

19. The computer program product of claim 17 , wherein implementing Non-Uniform Rational B Splines (NURBS);
based on the implementation of NURBS, producing a mathematical representation of a 3-D geometry corresponding to a physical head of a physical subject; and
generating the reference image based on the 3-D geometry.

Assignees

• Sony Interactive Entertainment America Llc
• Soul Machines Ltd

Inventors

• Sagar Mark Andrew
• Wu Tim Szu-Hsien
• Bonniwell Frank Filipe
• Alkouh Homoud B
• Hodges Colin Joseph

Classifications

• G06T13/40 Primary

  of characters, e.g. humans, animals or virtual beings · CPC title

• G06T7/251

  involving models · CPC title

• G06T7/75

  involving models · CPC title

• G06T13/00

  Animation · CPC title

• G06T15/04

  Texture mapping · CPC title