Region-based stabilized face tracking

Patent metadata

Field	Value
Publication number	US-12299905-B2
Application number	US-202318235563-A
Country	US
Kind code	B2
Filing date	Aug 18, 2023
Priority date	Jan 23, 2018
Publication date	May 13, 2025
Grant date	May 13, 2025

Abstract

Aspects of the present disclosure involve a system comprising a computer-readable storage medium storing at least one program and a method for accessing a set of images depicting at least a portion of a face. A set of facial regions of the face is identified, each facial region of the set of facial regions intersecting another facial region with at least one common vertex that is a member of a set of facial vertices. For each facial region of the set of facial regions, a weight formed from a set of region coefficients is generated. Based on the set of facial regions and the weight of each facial region of the set of facial regions, the face is tracked across the set of images.

First claim

What is claimed is:

1. A method comprising:
identifying a facial area depicted within a set of images within a video stream;
extracting a dense motion flow of the facial area from the set of images;
mapping the dense motion flow of each facial vertex within a set of facial regions in the facial area;
identifying highly correlated points within the set of facial regions;
computing correlative values of correlation matrices between vertices of the facial area corresponding to the identified highly correlated points, the computing of the correlative values comprising generating normalized correlation matrices for each facial vertex and averaging the correlation matrices into an averaged overall correlation matrix; and
tracking a face across the set of images based on the set of facial regions, the correlative values between the vertices, and the dense motion flow mapped to the respective facial region in the set of facial regions.

2. The method of claim 1 , further comprising:
computing inter-vertex distances on a mesh representing the facial area using an isomap process to form a distance matrix, the distance matrix being combined with the overall correlation matrix by normalizing the distance matrix and incorporating the distance matrix and the overall correlation matrix into an affinity matrix; and
solving for rigid pose parameters and expression parameters in alternating optimizations, such that during rigid pose optimization, the expression parameters are fixed while the rigid pose parameters are determined and, during expression optimization, the rigid pose parameters are fixed while the expression parameters are determined.

3. The method of claim 2 , further comprising:
based on the rigid pose parameters and the expression parameters, tracking a first region of the set of regions based on a first model and a second region of the set of regions based on a second model; and
overlaying a virtual object over a portion of the face depicted in the image in accordance with tracking the face across the set of images using the first and second models.

4. The method of claim 1 , further comprising:
identifying the set of facial regions of the face, each facial region of the set of facial regions intersecting another facial region with at least one common vertex which is a member of a set of facial vertices; and
adaptively modifying weights generated for each facial region to prioritize tracking of the face based on the weights.

5. The method of claim 1 , wherein identifying the set of facial regions comprises segmenting each facial region in the set of facial regions separately, and for each facial region of the set of facial regions, generating a weight formed from a set of region coefficients, wherein the face is tracked based on the set of facial regions and the weight of each facial region of the set of facial regions.

6. The method of claim 1 , wherein tracking the face comprises applying rigid and non-rigid optimizations to jointly estimate model and rigid pose parameters.

7. The method of claim 1 , further comprises:
determining that a first facial region in the set of regions corresponds to a first portion of the face that is more flexible than a second portion of the face corresponding to a second facial region in the set of regions.

8. The method of claim 7 , further comprising:
assigning a first weight to the first facial region and a second weight to the second facial region; and
adjusting the first weight to be greater than the second weight for optimizing an expression of the face, wherein the first weight and the second weight are computed using training data in an offline training process.

9. The method of claim 7 , further comprising:
assigning a first weight to the first facial region and a second weight to the second facial region; and
adjusting the second weight to be greater than the first weight for optimizing a head pose corresponding to the face, wherein the first weight and the second weight are computed using training data in an offline training process.

10. The method of claim 7 , wherein the first portion of the face comprises at least one of a cheek, mouth, or eye.

11. The method of claim 8 , further comprising determining values for first and second weights based on the training data comprising a plurality of synthetic facial animation sequences and first and second rigid transformations of the plurality of synthetic facial animation sequences.

12. The method of claim 11 , wherein the first transformation comprises a transformation captured from a video and the second transformation comprises a transformation captured from a static image.

13. The method of claim 1 , wherein a set of region coefficients comprises a rigid coefficient and a non-rigid coefficient, further comprising:
computing the rigid coefficient for each facial region while maintaining the non-rigid coefficient at a first fixed value; and
computing the non-rigid coefficient for each facial region while maintaining the rigid coefficient at a second fixed value.

14. The method of claim 1 , further comprising animating an avatar in accordance with tracking the face across the set of images.

15. A system comprising:
one or more processors; and
a non-transitory processor-readable storage medium storing processor executable instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising:
identifying a facial area depicted within a set of images within a video stream;
extracting a dense motion flow of the facial area from the set of images;
mapping the dense motion flow of each facial vertex within a set of facial regions in the facial area; and
identifying highly correlated points within the set of facial regions;
computing correlative values of correlation matrices between vertices of the facial area corresponding to the identified highly correlated points, the computing of the correlative values comprising generating normalized correlation matrices for each facial vertex and averaging the correlation matrices into an averaged overall correlation matrix; and
tracking a face across the set of images based on the set of facial regions, the correlative values between the vertices, and the dense motion flow mapped to the respective facial region in the set of facial regions.

16. The system of claim 15 , wherein the operations further comprise overlaying a graphical object over a portion of the face in accordance with tracking the face across the set of images, the operations comprising:
computing an expression magnitude for each region in the set of facial regions, the expression magnitude measuring a total non-rigid coefficient magnitude excluding a neutral expression.

17. The system of claim 16 , wherein the graphical object comprises virtual makeup.

18. The system of claim 17 , the operations further comprising animating an avatar in accordance with tracking the face across the set of images.

19. A non-transitory computer-readable medium comprising instructions that, when executed by at least one processor, configure the at least one processor to perform operations comprising:
identifying a facial area depicted within a set of images within a video stream;
extracting a dense motion flow of the facial area from the set of images;
mapping the dense motion flow of each facial vertex within a set of facial regions in the facial area; and
identifying highly correlated points within the set of facial regions;
computing correlative values of correlati

Assignees

• Snap Inc

Inventors

• Cao Chen
• Chai Menglei
• Luo Linjie
• Woodford Oliver

Classifications

• G06V40/176

  Dynamic expression · CPC title

• G06V40/165

  using facial parts and geometric relationships · CPC title

• G06V10/62

  relating to a temporal dimension, e.g. time-based feature extraction; Pattern tracking · CPC title

• G06V10/774

  Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting · CPC title

• G06V40/161

  Detection; Localisation; Normalisation · CPC title