Facial animation transfer

Patent metadata

Field	Value
Publication number	US-2023252704-A1
Application number	US-202318136470-A
Country	US
Kind code	A1
Filing date	Apr 19, 2023
Priority date	Jun 1, 2020
Publication date	Aug 10, 2023
Grant date	—

Abstract

Systems and methods are disclosed for generating, a source image sequence using an image sensor of the computing device, the source image sequence comprising a plurality of source images depicting a head and face, identifying driving image sequence data to modify face image feature data in the source image sequence, generating, using an image transformation neural network, a modified source image sequence comprising a plurality of modified source images depicting modified versions of the head and face, and storing the modified source image sequence on the computing device.

First claim

What is claimed is:

1 . A method comprising:
generating, by a computing device, a source image sequence using an image sensor of the computing device, the source image sequence comprising a plurality of source images depicting a source head and source face; identifying driving image sequence data to modify face image feature data in the source image sequence, the driving image sequence data comprising an ordered set of image arrays that depicts a head in different head poses; identifying an expression dataset to modify face image feature data in the source image sequence, the expression dataset comprising an unordered set of image arrays that depicts the head in different head poses and a face in different expressions; generating, using an image transformation neural network, a modified source image sequence comprising a plurality of modified source images depicting modified versions of the source head and source face based on the driving image sequence and the expression dataset; and storing the modified source image sequence on the computing device.

2 . The method of claim 1 , wherein each image array in the ordered set of image arrays depicts the head in a same pose from different viewpoints.

3 . The method of claim 1 , wherein each image array in the unordered set of image arrays depicts the face in a same expression from different viewpoints.

4 . The method of claim 1 , wherein the image transformation neural network comprises a keypoint detector neural network, a dense motion neural network and a generation neural network.

5 . The method of claim 4 , wherein the keypoint detector neural network is trained to identify a first set of keypoints for centers of a pair of eyes and a second set of keypoints for mouth corners.

6 . The method of claim 1 , wherein the image transformation neural network is trained to identify, for each image in the source image sequence, a driving image from the driving image sequence data based on the driving image having a similar head pose to the image in the source image sequence and an expression image from the expression dataset having a similar head pose and a similar expression to the image in the source image sequence, the identified driving image and the identified expression image being implemented by the image transformation neural network to modify a corresponding source image in the source image sequence using motion estimation differences between the identified driving image and the identified expression image.

7 . The method of claim 6 , wherein the image transformation neural network is configured to generate motion estimations differences between the identified driving image and the identified expression image.

8 . The method of claim 6 , wherein a first viewpoint of the identified driving image coincides with a second viewpoints of the identified expression image.

9 . A computing system, the computing system comprising:
one or more processors; and a memory storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising: generating, by a computing device, a source image sequence using an image sensor of the computing device, the source image sequence comprising a plurality of source images depicting a source head and source face; identifying driving image sequence data to modify face image feature data in the source image sequence, the driving image sequence data comprising an ordered set of image arrays that depicts a head in different head poses; identifying an expression dataset to modify face image feature data in the source image sequence, the expression dataset comprising an unordered set of image arrays that depicts the head in different head poses and a face in different expressions; generating, using an image transformation neural network, a modified source image sequence comprising a plurality of modified source images depicting modified versions of the source head and source face based on the driving image sequence and the expression dataset; and storing the modified source image sequence on the computing device.

10 . The computing system of claim 9 , wherein the modified source image sequence is transmitted as an ephemeral message to a second computing device.

11 . The computing system of claim 9 , wherein the instructions further configure the system to:
cause display of the modified source image sequence on a graphical user interface of the computing device.

12 . The computing system of claim 9 , wherein the image transformation neural network comprises a keypoint detector neural network, a dense motion neural network and a generation neural network.

13 . The computing system of claim 12 , wherein the keypoint detector neural network is trained to identify a first set of keypoints for centers of a pair of eyes and a second set of keypoints for mouth corners.

14 . The computing system of claim 9 , wherein the image transformation neural network is trained to identify, for each image in the source image sequence, a driving image from the driving image sequence data based on the driving image having a similar head pose to the image in the source image sequence and an expression image from the expression dataset having a similar head pose and a similar expression to the image in the source image sequence, the identified driving image and the identified expression image being implemented by the image transformation neural network to modify a corresponding source image in the source image sequence using motion estimation differences between the identified driving image and the identified expression image.

15 . The computing system of claim 14 , wherein the image transformation neural is configured to generate the motion estimations differences between the identified driving image and the identified expression image.

16 . The computing system of claim 14 , wherein a first viewpoint of the identified driving image coincides with a second viewpoints of the identified expression image.

17 . A non-transitory computer-readable storage medium storing instructions that when executed by one or more processors of a machine, cause the computer-readable storage medium to perform operations comprising:
generating, by a computing device, a source image sequence using an image sensor of the computing device, the source image sequence comprising a plurality of source images depicting a source head and source face; identifying driving image sequence data to modify face image feature data in the source image sequence, the driving image sequence data comprising an ordered set of image arrays that depicts a head in different head poses; identifying an expression dataset to modify face image feature data in the source image sequence, the expression dataset comprising an unordered set of image arrays that depicts the head in different head poses and a face in different expressions; generating, using an image transformation neural network, a modified source image sequence comprising a plurality of modified source images depicting modified versions of the source head and source face based on the driving image sequence and the expression dataset; and storing the modified source image sequence on the computing device.

18 . The computer-readable storage medium of claim 17 , wherein the modified source image sequence is transmitted as an ephemeral message to a second computing device.

19 . The computer-readable storage medium of claim 17 , wherein the instructions further configure the com

Assignees

• Snap Inc

Inventors

• Demyanov Sergey
• Podkin Aleksei
• Siarohin Aliaksandr
• Stoliar Aleksei
• Tulyakov Sergey

Classifications

• G06T13/00 Primary

  Animation · CPC title

• G06N3/08

  Learning methods · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G06V40/174

  Facial expression recognition · CPC title

• G06V40/171

  Local features and components; Facial parts (eye characteristics G06V40/18); Occluding parts, e.g. glasses; Geometrical relationships · CPC title