Techniques for re-aging faces in images and video frames

Patent metadata

Field	Value
Publication number	US-12307616-B2
Application number	US-202117473232-A
Country	US
Kind code	B2
Filing date	Sep 13, 2021
Priority date	Sep 13, 2021
Publication date	May 20, 2025
Grant date	May 20, 2025

Abstract

Techniques are disclosed for re-aging images of faces and three-dimensional (3D) geometry representing faces. In some embodiments, an image of a face, an input age, and a target age, are input into a re-aging model, which outputs a re-aging delta image that can be combined with the input image to generate a re-aged image of the face. In some embodiments, 3D geometry representing a face is re-aged using local 3D re-aging models that each include a blendshape model for finding a linear combination of sample patches from geometries of different facial identities and generating a new shape for the patch at a target age based on the linear combination. In some embodiments, 3D geometry representing a face is re-aged by performing a shape-from-shading technique using re-aged images of the face captured from different viewpoints, which can optionally be constrained to linear combinations of sample patches from local blendshape models.

First claim

What is claimed is:

1. A computer-implemented method for re-aging a face included in a first image, the method comprising:
generating an input age image corresponding to an input age associated with the face in the first image and a target age image associated with a target age of the face in the first image, wherein the input age image includes a first plurality of pixels having age values indicating the input age and the target age image includes a second plurality of pixels having age values indicating the target age, and wherein the first image, the input age image, and the target age image have the same spatial resolution;
generating, via a machine learning model, a second image based on a multi-channel tensor provided to the machine learning model at execution, wherein the multi-channel tensor comprises (i) the first image that includes the face, (ii) an input age image, and (iii) a target age image, wherein the second image includes one or more differences from the first image; and
combining the first image and the second image into a third image.

2. The computer-implemented method of claim 1 , further comprising:
training the machine learning model based on a first set of images of a plurality of facial identities at a plurality of ages.

3. The computer-implemented method of claim 2 , further comprising generating the first set of images by:
generating, via a first pre-trained machine learning model, a second set of images of the plurality of facial identities; and
generating, via a second pre-trained machine learning model, the first set of images based on the second set of images.

4. The computer-implemented method of claim 2 , wherein training the machine learning model comprises minimizing a loss function that comprises a L1 loss, a perceptual loss, and an adversarial loss.

5. The computer-implemented method of claim 1 , wherein the machine learning model comprises a U-Net architecture.

6. The computer-implemented method of claim 1 , further comprising:
generating a segmented image indicating a plurality of pixels included in the first image that correspond to skin of the face,
wherein the one or more pixels indicating the input age in the fourth image and the one or more pixels indicating the target age in the fifth image correspond to one or more pixels in the plurality of pixels included in the first image.

7. The computer-implemented method of claim 1 , wherein the fifth image further comprises one or more pixels indicating at least one other target age.

8. The computer-implemented method of claim 1 , wherein the target age is injected via a layer modulation technique.

9. The computer-implemented method of claim 1 , further comprising:
determining a deformation of three-dimensional geometry representing the face based on the third image.

10. The computer-implemented method of claim 1 , further comprising training a second machine learning model based on the third image, wherein, subsequent to training, the second machine learning model generates an output image including a given face at a given target age based on an input image of the given face at a given input age.

11. The computer-implemented method of claim 10 , wherein a 3D geometry representing the given face at the given input age is deformed based on the output image to generate a re-aged 3D geometry.

12. One or more non-transitory computer-readable storage media including instructions that, when executed by one or more processing units, cause the one or more processing units to perform steps for re-aging a face included in a first image, the steps comprising:
generating an input age image corresponding to an input age associated with the face in the first image and a target age image associated with a target age of the face in the first image, wherein the input age image includes a first plurality of pixels having age values indicating the input age and the target age image includes a second plurality of pixels having age values indicating the target age, and wherein the first image, the input age image, and the target age image have the same spatial resolution;
generating, via a machine learning model, a second image based on a multi-channel tensor provided to the machine learning model at execution, wherein the multi-channel tensor comprises (i) the first image that includes the face, (ii) an input age image, and (iii) a target age image, wherein the second image includes one or more differences from the first image; and
combining the first image and the second image into a third image.

13. The one or more non-transitory computer-readable storage media of claim 12 , wherein the instructions, when executed by the one or more processing units, further cause the one or more processing units to perform steps comprising:
generating a first set of images of a plurality of facial identities at a plurality of ages by:
generating, via a first pre-trained machine learning model, a second set of images of the plurality of facial identities, and
generating, via a second pre-trained machine learning model, the first set of images based on the second set of images; and

training the machine learning model based on the first set of images.

Assignees

• Disney Entpr Inc
• Eth Zuerich Eidgenoessische Technische Hochschule Zuerich

Inventors

• Zoss Gaspard
• Bradley Derek Edward
• Chandran Prashanth
• Urnau Gotardo Paulo Fabiano
• Sifakis Eftychios

Classifications

• G06T2219/2021

  Shape modification · CPC title

• G06T2207/20081

  Training; Learning · CPC title

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title

• G06T2207/20212

  Image combination · CPC title

• G06T2207/30201

  Face · CPC title