Geometry-guided controllable 3D head synthesis

Patent metadata

Field	Value
Publication number	US-12380630-B2
Application number	US-202318165794-A
Country	US
Kind code	B2
Filing date	Feb 7, 2023
Priority date	Feb 7, 2023
Publication date	Aug 5, 2025
Grant date	Aug 5, 2025

Abstract

Technologies are described and recited herein for producing controllable synthesized images include a geometry guided 3D GAN framework for high-quality 3D head synthesis with full control on camera poses, facial expressions, head shape, articulated neck and jaw poses; and a semantic SDF (signed distance function) formulation that defines volumetric correspondence from observation space to canonical space, allowing full disentanglement of control parameters in 3D GAN training.

First claim

What is claimed:

1. A method for generating controllable three-dimensional (3D) synthesized images, comprising:
conditioning, by a 3D model generator, 3D representations of a head geometry in a canonical space based on feature vectors and camera viewing parameters by:
mapping control parameters for the head geometry, corresponding to an input, and a signed distance function (SDF) onto the head geometry to produce 3D point-to-point volumetric correspondences of points in an observation space to the head geometry in the canonical space;
combining the 3D representations of the head geometry and the 3D point-to-point volumetric correspondences of points in the observation space to the head geometry in the canonical space to produce a 3D object; and
synthesizing the 3D object to combine feature layers of the 3D object, wherein the 3D object is produced as a layered combination of an inner feature layer and an external feature layer, the inner feature layer corresponding to the 3D point-to-point volumetric correspondences of points in the observation space, and the external feature layer corresponding to the 3D representations of the head geometry.

2. The method of claim 1 , further comprising:
volume rendering the synthesized 3D object based on the 3D point-to-point volumetric correspondences to produce a volume-rendered feature map;
synthesizing, by a super-resolution module, the volume-rendered feature map to produce a high-resolution image; and
encoding the high-resolution image to produce a controllable 3D model having the control parameters corresponding to the input.

3. The method of claim 2 , wherein the volume rendering, synthesizing by the super-resolution module, and the encoding are executed by a 3D GAN framework.

4. The method of claim 1 , further comprising determining, by a discriminator, whether the high-resolution image is authentic.

5. The method of claim 1 , wherein the mapping is executed by a trained multilayer perceptron (MLP).

6. The method of claim 5 , wherein the MLP is trained on 3D geometric information from a FLAME (faces learned with an articulated model and expressions) model to produce a volumetric correspondence map from the observation space to the canonical space.

7. The method of claim 1 , wherein the control parameters correspond to a FLAME (faces learned with an articulated model and expressions) model.

8. The method of claim 1 , wherein the control parameters include parameters for facial shape, facial expressions, and jaw and neck poses.

9. The method of claim 1 , further comprising optimizing a geometry prior loss to reduce a difference between a synthesized neural density field and a density field of the SDF.

10. The method of claim 1 , further comprising optimizing a control loss to control matching of synthesized images and an input control code upon encoding.

11. An image synthesis framework to produce controllable three-dimensional (3D) synthesized images, comprising:
a 3D model generator to generate 3D representations of a head geometry in a canonical space based on feature vectors and camera viewing parameters;
a multilayer perceptron (MLP) to map control parameters for the head geometry, corresponding to an input, and a signed distance function (SDF) onto the head geometry to produce 3D point-to-point volumetric correspondences of points in an observation space to the head geometry in the canonical space;
a first synthesizer to combine the 3D representations of the head geometry and the 3D point-to-point volumetric correspondences of points in the observation space to the head geometry in the canonical space to produce a 3D object;
a second synthesizer to synthesize the 3D object to combine feature layers of the 3D object, wherein the 3D object is produced as a layered combination of an inner feature layer and an external feature layer, the inner feature layer corresponding to the 3D point-to-point volumetric correspondences of points in the observation space, and the external feature layer corresponding to the 3D representations of the head geometry;
a renderer to volume render the synthesized 3D object based on the 3D point-to-point volumetric correspondences to produce a volume-rendered feature map;
a super resolution module to synthesize the volume-rendered feature map to produce a high-resolution image; and
an encoder to encode the high-resolution image to produce a controllable 3D model having the control parameters corresponding to the input.

12. The image synthesis framework of claim 11 , further comprising a discriminator to determine whether the high-resolution image is authentic.

13. The image synthesis framework of claim 11 , further comprising an encoder to reduce a difference between a synthesized neural density field and a density field of the SDF.

14. The image synthesis framework of claim 11 , further comprising an encoder to control matching of synthesized images and an input control code.

Assignees

• Lemon Inc

Inventors

• Xu Hongyi
• Song Guoxian
• Jiang Zihang
• Zhang Jianfeng
• Shi Yichun
• Liu Jing
• Ma Wanchun
• Feng Jiashi
• Luo Linjie

Classifications

• G06T3/4053 Primary

  based on super-resolution, i.e. the output image resolution being higher than the sensor resolution · CPC title

• G06V40/176

  Dynamic expression · CPC title

• G06T3/4046

  using neural networks · CPC title

• G06T17/00

  Three-dimensional [3D] modelling for computer graphics · CPC title

• G06T2219/2008

  Assembling, disassembling · CPC title