Geometry-aware three-dimensional synthesis in all angles

Patent metadata

Field	Value
Publication number	US-12586303-B2
Application number	US-202318165619-A
Country	US
Kind code	B2
Filing date	Feb 7, 2023
Priority date	Feb 7, 2023
Publication date	Mar 24, 2026
Grant date	Mar 24, 2026

Abstract

A three-dimensional generative adversarial network includes a generator, a discriminator, and a renderer. The generator is configured to receive an intermediate latent code mapped from a latent code and a camera pose, generate two-dimensional backgrounds for a set of images, and generate, based on the intermediate latent code, multi-grid representation features. The renderer is configured to synthesize images based on the camera pose, a camera pose offset, and the multi-grid representation features; the camera pose offset being mapped from the latent code and the camera pose; and render a foreground mask. The discriminator is configured to supervise a training of the foreground mask with an up-sampled image and a super-resolved image.

First claim

What is claimed is:

1 . A method for generating three-dimensional (3D) synthesized images, the method comprising:
pre-processing a first set of images and a second set of images by:
scaling the first set of images to a predetermined size and aligning the first set of images at a center of a reference;
centering a bounding box at the reference for the second set of images; and
adjusting a scale and a translation of the center of the reference with constant offsets for the second set of images; and
tuning a volume rendering for each image of the first and second sets of images by:
associating each image with a latent code and synthesizing the latent code at a view of a camera pose;
mapping a camera pose offset from the latent code and the camera pose;
generating synthesized images based on the camera pose and the camera pose offset; and

disentangling a foreground from a background of the synthesized images by:
generating two-dimensional backgrounds for the first and second sets of images;
determining a foreground mask during the volume rendering; and
supervising, by a discriminator, a training of the foreground mask with an up-sampled image and a super-resolved image.

2 . The method of claim 1 , further comprising:
cropping the second set of images at the adjusted scale and the center of the reference.

3 . The method of claim 1 , further comprising:
composing an image based on the foreground mask; and generating a super-resolved foreground mask based on the composed image.

4 . The method of claim 1 , further comprising:
augmenting a tri-plane with a depth dimension.

5 . The method of claim 1 , further comprising:
generating 3D portraits from the latent code.

6 . The method of claim 1 , further comprising:
receiving a single image; and generating 3D portraits from the single image.

7 . The method of claim 6 , further comprising:
performing an optimization to determine an optimized latent code for the single image; and altering generator parameters with the optimized latent code.

8 . The method of claim 1 , wherein the reference is a head of a person in the first and second sets of images, the first set of images are frontal images with facial landmarks, and the second set of images are back head images or large-pose images.

9 . The method of claim 1 , wherein the latent code is sampled from a Gaussian distribution.

10 . The method of claim 1 , wherein the camera pose is estimated from a set of training images.

11 . A non-transitory, computer-readable medium storing one or more instructions executable by a computer system to perform one or more operations, comprising:
pre-processing a first set of images and a second set of images by:
scaling the first set of images to a predetermined size and aligning the first set of images at a center of a reference;
centering a bounding box at the reference for the second set of images; and
adjusting a scale and a translation of the center of the reference with constant offsets for the second set of images; and
tuning a volume rendering for each image of the first and second sets of images by:
associating each image with a latent code and synthesizing the latent code at a view of a camera pose;
mapping a camera pose offset from the latent code and the camera pose;
generating synthesized images based on the camera pose and the camera pose offset; and

disentangling a foreground from a background of the synthesized images by:
generating two-dimensional backgrounds for the first and second sets of images;
determining a foreground mask during the volume rendering; and
supervising, by a discriminator, a training of the foreground mask with an up-sampled image and a super-resolved image.

12 . The medium of claim 11 , the operations further comprising:
cropping the second set of images at the adjusted scale and the center of the reference.

13 . The medium of claim 11 , the operations further comprising:
composing an image based on the foreground mask; and generating a super-resolved foreground mask based on the composed image.

14 . The medium of claim 11 , the operations further comprising:
augmenting a tri-plane with a depth dimension.

15 . The medium of claim 11 , the operations further comprising:
generating 3D portraits from the latent code.

16 . The medium of claim 11 , the operations further comprising:
receiving a single image; and generating 3D portraits from the single image.

17 . The medium of claim 16 , the operations further comprising:
performing an optimization to determine an optimized latent code for the single image; and altering generator parameters with the optimized latent code.

18 . The medium of claim 11 , wherein the reference is a head of a person in the first and second sets of images, the first set of images are frontal images with facial landmarks, and the second set of images are back head images or large-pose images.

19 . A system comprising:
one or more computers; and one or more computer memory devices interoperably coupled with the one or more computers and having computer-readable storage media storing one or more instructions that, when executed by the one or more computers, perform one or more operations comprising: pre-processing a first set of images and a second set of images by: scaling the first set of images to a predetermined size and aligning the first set of images at a center of a reference; centering a bounding box at the reference for the second set of images; and adjusting a scale and a translation of the center of the reference with constant offsets for the second set of images; and tuning a volume rendering for each image of the first and second sets of images by:
associating each image with a latent code and synthesizing the latent code at a view of a camera pose;
mapping a camera pose offset from the latent code and the camera pose;
generating synthesized images based on the camera pose and the camera pose offset; and
disentangling a foreground from a background of the synthesized images by:
generating two-dimensional backgrounds for the first and second sets of images;
determining a foreground mask during the volume rendering; and
supervising, by a discriminator, a training of the foreground mask with an up-sampled image and a super-resolved image.

20 . The system of claim 19 , the operations further comprising:
composing an image based on the foreground mask; and generating a super-resolved foreground mask based on the composed image.

Assignees

• Lemon Inc

Inventors

• Xu Hongyi
• An Sizhe
• Shi Yichun
• Song Guoxian
• Luo Linjie

Classifications

• G06T11/00 Primary

  Two-dimensional [2D] image generation · CPC title

• G06T2219/2016

  Rotation, translation, scaling · CPC title

• G06T2219/2004

  Aligning objects, relative positioning of parts · CPC title

• G06T2207/30196

  Human being; Person · CPC title

• G06T2207/20081

  Training; Learning · CPC title