Generating a layered animatable puppet using a content stream

Patent metadata

Field	Value
Publication number	US-10489959-B2
Application number	US-201715786558-A
Country	US
Kind code	B2
Filing date	Oct 17, 2017
Priority date	Jun 6, 2017
Publication date	Nov 26, 2019
Grant date	Nov 26, 2019

Abstract

Certain embodiments involve automatically generating a layered animatable puppet using a content stream. For example, a system identifies various frames of a content stream that includes a character performing various gestures usable for generating a layered puppet. The system separates the various frames of the content stream into various individual layers. The system extracts a face of the character from the various individual layers and creates the layered puppet by combining the individual layers and using the face of the character. The system can output the layered puppet for animation to perform a gesture of the various gestures.

First claim

The invention claimed is:

1. A method for generating a layered animatable puppet, the method comprising:
accessing, by a processor, a plurality of frames of a content stream depicting a character in motion;
identifying, by the processor, a subset of frames from the plurality of frames, the subset including (i) a first frame from the plurality of frames depicting the character performing a first gesture and (ii) a second frame from the plurality of frames depicting the character performing a second gesture different from the first gesture;
extracting, by the processor, a face of the character from the subset of frames;
generating, by the processor, (i) a first augmented layer by applying a style or texture to color information of the face as extracted from the first frame and (ii) a second augmented layer by applying the style or texture to color information of the face as extracted from the second frame;
creating, by the processor, a layered puppet by combining the first augmented layer and the second augmented layer; and
outputting, by the processor, the layered puppet for animation to perform a plurality of gestures.

2. The method of claim 1 , further comprising:
generating, by the processor, a plurality of augmented layers that includes the first augmented layer and the second augmented layer by performing operations comprising:
receiving, by the processor, user input indicating a command to simulate a model of the face;
generating, by the processor, a three-dimensional model of a face from a particular frame of the subset of frames; and
rotating, by the processor, the three-dimensional model to generate a plurality of views of the face of the character to generate one or more of the plurality of augmented layers having a simulated pose, view, or angle of the character.

3. The method of claim 1 , further comprising:
generating, by the processor, a plurality of augmented layers that includes the first augmented layer and the second augmented layer by performing operations comprising:
receiving, by the processor, user input indicating a command to caricaturize one or more frames from the subset of frames; and
generating, by the processor, one or more of the plurality of augmented layers by warping or distorting the one or more frames from the subset of frames in response to receiving the command.

4. The method of claim 3 , wherein warping or distorting the one or more frames from the subset of frames comprises:
identifying, by the processor, a feature of the character in an individual frame from the subset of frames;
determining, by the processor, a characteristic of the feature of the character; and
warping or distorting, by the processor, the feature of the character based on the characteristic.

5. The method of claim 1 , wherein identifying the subset of frames comprises:
outputting, by the processor and to a user device, a request to perform a particular gesture;
analyzing, by the processor, the content stream to determine that a gesture performed by the character in a frame of the plurality of frames corresponds to the particular gesture; and
including, by the processor, the frame in the subset of frames in response to determining that the gesture of the plurality of gestures corresponds to the particular gesture.

6. The method of claim 1 , wherein identifying the subset of frames comprises:
analyzing, by the processor, audio data corresponding to the content stream to identify a frame of the content stream that is aligned with a gesture that the character is making, wherein the gesture comprises a phoneme or a sound.

7. The method of claim 1 , further comprising identifying a feature of the character in the content stream by:
identifying, by the processor, the feature based on a mask of the feature; and
extracting, by the processor, the face of the character using the identified feature.

8. The method of claim 1 , further comprising:
generating, by the processor, a temporal guide for generating a video, the video comprising the layered puppet and the temporal guide usable for determining an amount of temporal flickering in the video, wherein generating the temporal guide comprises advecting, by the processor and using an underlying motion field, an individual frame from the subset of frames or an augmented layer from a plurality of augmented layers that includes the first augmented layer and the second augmented layer;
controlling, by the processor, the amount of temporal flickering in the video by varying an amount of blur of the advected individual frame or advected augmented layer; and
outputting, by the processor, the video for display via a user interface.

9. The method of claim 1 , wherein extracting the face of the character from the subset of frames comprises:
applying soft masks, respectively, to the subset of frames; and
modifying transparency settings of each soft mask such that the soft mask is transparent in a frame region depicting the face and opaque in a frame region depicting background content other than the face.

10. A system comprising:
a processing device; and
a non-transitory computer-readable medium communicatively coupled to the processing device, wherein the processing device is configured to perform operations comprising:
accessing a plurality of frames of a content stream depicting a character in motion;
identifying a subset of frames from the plurality of frames, the subset including (i) a first frame from the plurality of frames depicting the character performing a first gesture and (ii) a second frame from the plurality of frames depicting the character performing a second gesture different from the first gesture;
extracting a face of the character from the subset of frames;
generating (i) a first augmented layer by applying a style or texture to color information of the face as extracted from the first frame and (ii) a second augmented layer by applying the style or texture to color information of the face as extracted from the second frame;
creating a layered puppet by combining the first augmented layer and the second augmented layer and using the face of the character; and
outputting the layered puppet for animation to perform a plurality of gestures.

11. The system of claim 10 , the operations further comprising generating a plurality of augmented layers that includes the first augmented layer and the second augmented layer by performing additional operations comprising:
receiving user input indicating a command to simulate a model of the face;
generating a three-dimensional model of a face from a particular frame of the subset of frames; and
rotating the three-dimensional model to generate a plurality of views of the face of the character to generate one or more of the plurality of augmented layers having a simulated pose, view, or angle of the character.

12. The system of claim 10 , the operations further comprising generating a plurality of augmented layers that includes the first augmented layer and the second augmented layer by performing additional operations comprising:
receiving user input indicating a command to caricaturize one or more frames from the subset of frames; and
generating one or more of the plurality of augmented layers by warping or distorting the one or more frames from the subset of frames in response to receiving the command.

13. The system of claim 12 , wherein warping or distorting the one or more frames from the subset of frames comprises:
identifying a feature of the character in an individual frame from the subset of frames;
determining a characteristic of the feature of the character; and
warping or distorting the fea

Assignees

• Adobe Inc

Inventors

• Simons David
• Fiser Jakub

Classifications

• G06T11/10

  Texturing; Colouring; Generation of textures or colours (retouching, inpainting or scratch removal G06T5/77) · CPC title

• G06T15/04

  Texture mapping · CPC title

• G06T2207/20221

  Image fusion; Image merging · CPC title

• G06T13/40 Primary

  of characters, e.g. humans, animals or virtual beings · CPC title

• G06T13/80

  Two-dimensional [2D] animation, e.g. using sprites · CPC title