Synthesizing sequences of images for movement-based performance

Patent metadata

Field	Value
Publication number	US-12243140-B2
Application number	US-202117526647-A
Country	US
Kind code	B2
Filing date	Nov 15, 2021
Priority date	Nov 15, 2021
Publication date	Mar 4, 2025
Grant date	Mar 4, 2025

Abstract

A technique for rendering an input geometry includes generating a first segmentation mask for a first input geometry and a first set of texture maps associated with one or more portions of the first input geometry. The technique also includes generating, via one or more neural networks, a first set of neural textures for the one or more portions of the first input geometry. The technique further includes rendering a first image corresponding to the first input geometry based on the first segmentation mask, the first set of texture maps, and the first set of neural textures.

First claim

What is claimed is:

1. A computer-implemented method for rendering an input geometry, the computer-implemented method comprising:
generating a first segmentation mask for a first input three-dimensional (3D) geometry and a first plurality of texture maps associated with a plurality of portions of the first input 3D geometry;
generating, via execution of a plurality of generator blocks included in one or more neural networks, a first plurality of neural textures for the plurality of portions of the first input 3D geometry, wherein each generator block included in the plurality of generator blocks generates a neural texture for a different portion included in the plurality of portions of the first input 3D geometry, wherein portions of the first input 3D geometry corresponding to different generator blocks included in the plurality of generator blocks are non-overlapping; and
rendering a first image corresponding to the first input 3D geometry based on the first segmentation mask, the first plurality of texture maps, and the first plurality of neural textures.

2. The computer-implemented method of claim 1 , further comprising training the one or more neural networks based on a training dataset that includes a second plurality of texture maps and a plurality of segmentation masks for a plurality of synthetic geometries.

3. The computer-implemented method of claim 1 , further comprising training the one or more neural networks based on one or more predictions generated by a discriminator neural network from one or more images produced by the one or more neural networks.

4. The computer-implemented method of claim 1 , wherein generating the first segmentation mask and the first plurality of texture maps comprises:
deforming a template mesh to match the first input 3D geometry; and
generating the first segmentation mask and the first plurality of texture maps based on a pose associated with the first input 3D geometry.

5. The computer-implemented method of claim 1 , further comprising rendering a second image corresponding to a second input geometry based on a second segmentation mask for the second input geometry, a second plurality of texture maps associated with a plurality of portions of the second input geometry, and the first plurality of neural textures.

6. The computer-implemented method of claim 1 , wherein generating the first plurality of neural textures comprises, for each portion included in the plurality of portions of the first input 3D geometry:
generating an input vector based on sampling a distribution of latent variables associated with the portion; and
inputting the input vector into the generator block corresponding to the portion.

7. The computer-implemented method of claim 1 , wherein rendering the first image comprises:
sampling the first plurality of neural textures based on the first plurality of texture maps to generate a plurality of screen-space neural features;
generating a composited set of screen-space neural features based on the first segmentation mask and the plurality of screen-space neural features; and
applying one or more convolutional layers to the composited set of screen-space neural features to render the first image.

8. The computer-implemented method of claim 1 , wherein the one or more neural networks comprise a generative neural network.

9. The computer-implemented method of claim 1 , wherein the first input 3D geometry comprises a face.

10. The computer-implemented method of claim 1 , wherein the plurality of portions of the first input 3D geometry comprises at least one of a skin, a hair, one or more eyes, a mouth, or a background.

11. One or more non-transitory computer readable media storing instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of:
generating one or more maps associated with a plurality of portions of a first input three-dimensional (3D) geometry;
generating, via execution of a plurality of generator blocks included in one or more neural networks, a first plurality of neural textures for the plurality of portions of the first input 3D geometry, wherein each generator block included in the plurality of generator blocks generates a neural texture for a different portion included in the plurality of portions of the first input 3D geometry, wherein portions of the first input 3D geometry corresponding to different generator blocks included in the plurality of generator blocks are non-overlapping; and
rendering a first image corresponding to the first input 3D geometry based on the one or more maps and the first plurality of neural textures.

12. The one or more non-transitory computer readable media of claim 11 , wherein the instructions further cause the one or more processors to perform the step of training the one or more neural networks based on a training dataset that includes a plurality of maps associated with a plurality of synthetic geometries.

13. The one or more non-transitory computer readable media of claim 12 , wherein training the one or more neural networks comprises updating parameters of the one or more neural networks based on one or more predictions generated by a discriminator neural network from one or more images produced by the one or more neural networks.

14. The one or more non-transitory computer readable media of claim 11 , wherein generating the one or more maps comprises:
deforming a template mesh to match the first input 3D geometry; and
generating a segmentation mask and a set of texture maps based on a pose associated with the first input 3D geometry.

15. The one or more non-transitory computer readable media of claim 11 , wherein the instructions further cause the one or more processors to perform the step of rendering a second image corresponding to the first input 3D geometry based on the one or more maps and a second set of neural textures for the plurality of portions of the first input 3D geometry.

16. The one or more non-transitory computer readable media of claim 11 , wherein generating the first plurality of neural textures comprises:
generating one or more input vector based on sampling one or more distributions of latent variables associated with the plurality of portions;
executing a mapping network included in the one or more neural networks to convert one or more sampled vectors into one or more input vectors; and
executing the plurality of generator blocks to convert the one or more input vectors into the first plurality of neural textures.

17. The one or more non-transitory computer readable media of claim 11 , wherein rendering the first image comprises:
sampling the first plurality of neural textures based on a first plurality of texture maps included in the one or more maps to generate a plurality of screen-space neural features;
generating a composited set of screen-space neural features based on a first segmentation mask included in the one or more maps and the plurality of screen-space neural features; and
applying one or more convolutional layers to the composited set of screen-space neural features to render the first image.

18. The one or more non-transitory computer readable media of claim 11 , wherein the first input 3D geometry comprises a face.

19. The one or more non-transitory computer readable media of claim 11 , wherein the one or more maps comprises at least one of a skin texture map, a hair texture map, an eye texture map, a mouth texture map, or a background texture map.

20. A system, comprising:
one or more memo

Assignees

• Disney Entpr Inc
• Eth Zuerich Eidgenoessische Technische Hochschule Zuerich

Inventors

• Bradley Derek Edward
• Chandran Prashanth
• Urnau Gotardo Paulo Fabiano
• Zoss Gaspard

Classifications

• G06T15/04

  Texture mapping · CPC title

• G06T2207/30201

  Face · CPC title

• G06T2207/20081

  Training; Learning · CPC title

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title

• G06N3/08

  Learning methods · CPC title