Generating target-character-animation sequences based on style-aware puppets patterned after source-character-animation sequences

Patent metadata

Field	Value
Publication number	US-10789754-B2
Application number	US-201816047839-A
Country	US
Kind code	B2
Filing date	Jul 27, 2018
Priority date	Jul 27, 2018
Publication date	Sep 29, 2020
Grant date	Sep 29, 2020

Abstract

This disclosure relates to methods, non-transitory computer readable media, and systems that use style-aware puppets patterned after a source-character-animation sequence to generate a target-character-animation sequence. In particular, the disclosed systems can generate style-aware puppets based on an animation character drawn or otherwise created (e.g., by an artist) for the source-character-animation sequence. The style-aware puppets can include, for instance, a character-deformational model, a skeletal-difference map, and a visual-texture representation of an animation character from a source-character-animation sequence. By using style-aware puppets, the disclosed systems can both preserve and transfer a detailed visual appearance and stylized motion of an animation character from a source-character-animation sequence to a target-character-animation sequence.

First claim

We claim:

1. A system comprising:
at least one processor; and
at least one non-transitory computer memory comprising a source-character-animation sequence of a two-dimensional animation character portrayed in source drawings corresponding to a source-skeletal-animation sequence of a source skeleton in source-skeletal poses, and instructions that, when executed by at least one processor, cause the system to:
generate style-aware puppets for the source drawings portraying the two-dimensional animation character, each style-aware puppet comprising a coarse-deformation model of a source drawing portraying the two-dimensional animation character from the source-character-animation sequence, a residual-deformation model of the source drawing portraying the two-dimensional animation character, a skeletal-difference map comprising differences between a source-skeletal pose and a stylized-skeletal pose corresponding to the source drawing portraying the two-dimensional animation character, and a visual-texture representation of the source drawing portraying the two-dimensional animation character;
identify a target-skeletal-animation sequence comprising a target skeleton in target-skeletal poses;
generate target-style-aware puppets corresponding to the target-skeletal-animation sequence by, for a target-skeletal pose of the target-skeletal poses:
comparing the target-skeletal pose with the source-skeletal-animation sequence to select a set of source-skeletal poses;
identifying a set of source drawings from the source-character-animation sequence from the source-character-animation sequence corresponding to the set of source-skeletal poses;
blending the style-aware puppets corresponding to the set of source drawings to create a target-style-aware puppet; and

based on the target-style-aware puppets, generate a target-character-animation sequence of the two-dimensional animation character portrayed in target drawings corresponding to the target-skeletal-animation sequence.

2. The system of claim 1 , further comprising instructions that, when executed by the at least one processor, cause the system to:
compare the target-skeletal pose with the source-skeletal-animation sequence to select the set of source-skeletal poses by:
identifying a set of target-skeletal poses adjacent to the target-skeletal pose in the target-skeletal-animation sequence; and
selecting the set of source-skeletal poses by comparing trajectories of joints across the set of target-skeletal poses with trajectories of joints across the set of source-skeletal poses; and

identify the set of source drawings by identifying source drawings corresponding to the set of source-skeletal poses in the source-character-animation sequence.

3. The system of claim 1 , further comprising instructions that, when executed by the at least one processor, cause the system to blend the style-aware puppets corresponding to the set of source drawings by:
interpolating coarse-deformation models for the style-aware puppets corresponding to the set of source drawings to create a modified-coarse-deformation model for the target-style-aware puppet;
linearly blending residual-deformation models for the style-aware puppets corresponding to the set of source drawings to create a modified-residual-deformation model for the target-style-aware puppet;
linearly blending skeletal-difference maps for the style-aware puppets to create a modified-skeletal-difference map for the target-style-aware puppet; and
linearly blending visual-texture representations for the style-aware puppets corresponding to the set of source drawings to create a modified-visual-texture representation for the target-style-aware puppet.

4. The system of claim 3 , further comprising instructions that, when executed by the at least one processor, cause the system to linearly blend the visual-texture representations for the style-aware puppets corresponding to the set of source drawings by:
rectifying the visual-texture representations for each style-aware puppet corresponding to the set of source drawings based on the coarse-deformation model for the style-aware puppet and the residual-deformation model for the style-aware puppet; and
linearly blending the rectified visual-texture representations to create the modified-visual-texture representation for the target-style-aware puppet.

5. The system of claim 4 , further comprising instructions that, when executed by the at least one processor, cause the system to, based on the target-style-aware puppet, generate the target-character-animation sequence of the two-dimensional animation character portrayed in the target drawings by, for the target-style-aware puppet:
applying the modified-skeletal-difference map to the target-skeletal pose to generate a stylized-target-skeletal pose; and
utilizing the modified-coarse-deformation model for the target-style-aware puppet, the modified-residual-deformation model for the target-style-aware puppet, and the stylized-target-skeletal pose to generate a target drawing portraying the two-dimensional animation character.

6. The system of claim 4 , further comprising instructions that, when executed by the at least one processor, cause the system to, based on the target-style-aware puppet, generate the target-character-animation sequence of the two-dimensional animation character portrayed in the target drawings by, for the target-style-aware puppet, utilize a segmentation-guidance channel and a temporal-appearance-guidance channel with the modified-visual-texture representation to generate the two-dimensional animation character in the target drawing.

7. The system of claim 1 , further comprising instructions that, when executed by the at least one processor, cause the system to determine the coarse-deformation model of the source drawing portraying the two-dimensional animation character by:
generating a layered as-rigid-as-possible (“ARAP”) mesh for a layered-template puppet, wherein the layered ARAP mesh comprises ARAP mesh layers corresponding to semantic layers of the layered-template puppet; and
adjusting the ARAP mesh layers of the layered ARAP mesh to conform to the semantic layers of the layered-template puppet positioned according to the source drawing.

8. The system of claim 1 , further comprising instructions that, when executed by the at least one processor, cause the system to select the set of source drawings from the source-character-animation sequence by:
identifying a set of target-skeletal poses adjacent to the target-skeletal pose in the target-skeletal-animation sequence comprising the target skeleton in the target-skeletal poses;
selecting a set of source-skeletal poses analogous to the set of target-skeletal poses by comparing trajectories of joints across the set of target-skeletal poses with trajectories of joints across the set of source-skeletal poses; and
selecting the set of source drawings by identifying source drawings corresponding to the set of source-skeletal poses in the source-character-animation sequence.

9. The system of claim 8 , further comprising instructions that, when executed by the at least one processor, cause the system to blend the style-aware puppets corresponding to the set of source drawings to create the target-style-aware puppet by:
identifying additional sets of source drawings from the source-character-animation sequence, the additional sets of source drawings each comprising at least one source drawing overlapping with the set of source drawings in the source-character-animation sequence; and
blending style-aware puppets corresponding to the additional sets of source drawings with the style-aware puppets corresponding to the set of source drawings.

10. The syst

Assignees

• Adobe Inc
• Czech Technical Univ In Prague

Inventors

• Kim Vladimir
• Li Wilmot
• Dvoro{Hacek Over (Z)}{Hacek Over (N)}Ák Marek
• Sýkora Daniel

Classifications

• G06T13/80 Primary

  Two-dimensional [2D] animation, e.g. using sprites · CPC title

• G06T13/40 Primary

  of characters, e.g. humans, animals or virtual beings · CPC title