Generating an image including a source individual

Patent metadata

Field	Value
Publication number	US-12555273-B2
Application number	US-202318320117-A
Country	US
Kind code	B2
Filing date	May 18, 2023
Priority date	May 19, 2022
Publication date	Feb 17, 2026
Grant date	Feb 17, 2026

Abstract

One embodiment of the present invention sets forth a technique for performing face swapping. The technique includes generating a latent representation of a first facial identity included in an input image. The technique further includes identifying a first identity-specific neural network layer associated with a second facial identity from a plurality of identity-specific neural network layers, wherein each neural network layer included in the plurality of identity-specific neural network layers is associated with a different facial identity. The technique further includes executing the first identity-specific neural network layer and one or more other neural network layers to generate one or more decoder input values corresponding to the latent representation. The technique further includes executing a decoder neural network that converts the one or more decoder input values into an output image depicting the second facial identity.

First claim

What is claimed is:

1 . A computer-implemented method for performing face swapping, the computer-implemented method comprising:
generating a latent representation of a first facial identity included in an input image; identifying a first identity-specific neural network layer associated with a second facial identity from a plurality of identity-specific neural network layers, wherein each neural network layer included in the plurality of identity-specific neural network layers is associated with a different facial identity, and wherein identifying the first identity-specific neural network layer comprises replacing a previously-selected identity-specific neural network layer with the first identity-specific neural network layer; executing the first identity-specific neural network layer and one or more other neural network layers to generate one or more decoder input values corresponding to the latent representation; and executing a decoder neural network that converts the one or more decoder input values into an output image depicting the second facial identity.

2 . The computer-implemented method of claim 1 , wherein one of the plurality of identity-specific neural network layers is stored in memory and at least one second identity-specific neural network layer of the plurality of identity-specific neural network layers is stored in persistent storage.

3 . The computer-implemented method of claim 1 , wherein identifying the first identity-specific neural network layer comprises retrieving the first identity-specific neural network layer from persistent storage.

4 . The computer-implemented method of claim 3 , wherein retrieving the first identity-specific neural network layer from the persistent storage comprises searching the persistent storage for the second facial identity, wherein the first identity-specific neural network layer is associated with the second facial identity in the persistent storage.

5 . The computer-implemented method of claim 1 , wherein identifying the first identity-specific neural network layer comprises de-allocating memory used by the previously-selected identity-specific neural network layer.

6 . The computer-implemented method of claim 1 , wherein the first identity-specific neural network layer generates one or more latent space values based on the latent representation of the first facial identity, and
wherein the one or more other neural network layers generate the one or more decoder input values based on the one or more latent space values.

7 . The computer-implemented method of claim 1 , further comprising:
training one or more neural networks based on a loss associated with the input image and the output image, wherein the one or more neural networks include the first identity-specific neural network layer.

8 . The computer-implemented method of claim 7 , wherein the one or more neural networks being trained further include one or more decoder input generator neural network layers corresponding to the one or more other neural network layers and the decoder neural network.

9 . The computer-implemented method of claim 7 , wherein the loss comprises a reconstruction loss determined based on a difference between the input image and the output image.

10 . The computer-implemented method of claim 1 , wherein the first facial identity comprises at least one of a personal identity, an age, or a lighting condition.

11 . The computer-implemented method of claim 1 , wherein the one or more other neural network layers comprise one or more decoder input generator dense neural network layers that generate the one or more decoder input values.

12 . The computer-implemented method of claim 1 , wherein
the one or more decoder input values correspond to one or more adaptive instance normalization coefficients that modify convolution operations performed by the decoder neural network.

13 . One or more non-transitory computer-readable media storing instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of:
generating a latent representation of a first facial identity included in an input image; executing one or more identity-independent neural network layers to generate one or more decoder input values corresponding to the latent representation; executing one or more neural network layers associated with a second facial identity to generate one or more identity-specific decoder input values based on the one or more decoder input values; and executing a decoder neural network that converts the one or more identity-specific decoder input values into an output image depicting the second facial identity.

14 . The one or more non-transitory computer-readable media of claim 13 , wherein the one or more identity-independent neural network layers include:
an identity-independent neural network layer that generates one or more latent space values based on the latent representation of the first facial identity; and one or more decoder input generator neural network layers that generate the one or more decoder input values based on the one or more latent space values.

15 . The one or more non-transitory computer-readable media of claim 13 , wherein executing the decoder neural network comprise:
inputting a first portion of the one or more identity-specific decoder input values into a first decoder layer in the decoder neural network; inputting a second portion of the one or more identity-specific decoder input values into a second decoder layer in the decoder neural network; and executing the first decoder layer and the second decoder layer to generate the output image.

16 . The one or more non-transitory computer-readable media of claim 13 , further comprising:
training one or more neural networks based on a loss associated with the input image and the output image, wherein the one or more neural networks include the one or more neural network layers associated with the second facial identity.

17 . The one or more non-transitory computer-readable media of claim 13 , wherein executing the one or more neural network layers associated with the second facial identity comprises:
generating an identity-specific one-hot vector corresponding to the second facial identity; executing the one or more neural network layers associated with the second facial identity to generate one or more values based on the identity-specific one-hot vector; and generating, based on the one or more values, the one or more decoder input values corresponding to the latent representation.

18 . A system, comprising:
one or more memories that store instructions, and one or more processors that are coupled to the one or more memories and, when executing the instructions, are configured to perform the steps of:
generating a latent representation of a first facial identity included in an input image;
identifying a first identity-specific neural network layer associated with a second facial identity from a plurality of identity-specific neural network layers, wherein each neural network layer included in the plurality of identity-specific neural network layers is associated with a different facial identity, and wherein identifying the first identity-specific neural network layer comprises replacing a previously-selected identity-specific neural network layer with the first identity-specific neural network layer;
executing the first identity-specific neural network layer and one or more other neural network layers to generate one or more decoder input values correspo

Assignees

• Disney Entpr Inc
• Eth Zuerich Eidgenoessische Technische Hochschule Zuerich

Inventors

• Naruniec Jacek Krzysztof
• Kansy Manuel Jakob
• Mignone Graziana
• Schroers Christopher Richard
• Weber Romann Matthew

Classifications

• G06V10/82

  using neural networks · CPC title

• G06V10/776

  Validation; Performance evaluation · CPC title

• G06V40/178

  estimating age from face image; using age information for improving recognition · CPC title

• G06V40/172

  Classification, e.g. identification · CPC title

• G06V10/77

  Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation · CPC title