Techniques for enhancing skin renders using neural network projection for rendering completion

Patent metadata

Field	Value
Publication number	US-12482076-B2
Application number	US-202117536777-A
Country	US
Kind code	B2
Filing date	Nov 29, 2021
Priority date	Jan 28, 2021
Publication date	Nov 25, 2025
Grant date	Nov 25, 2025

Abstract

Techniques are disclosed for generating photorealistic images of head portraits. A rendering application renders a set of images that include the skin of a face and corresponding masks indicating pixels associated with the skin in the images. An inpainting application performs a neural projection technique to optimize a set of parameters that, when input into a generator model, produces a set of projection images, each of which includes a head portrait in which (1) skin regions resemble the skin regions of the face in a corresponding rendered image; and (2) non-skin regions match the non-skin regions in the other projection images when the rendered set of images are standalone images, or transition smoothly between consecutive projection images in the case when the rendered set of images are frames of a video. The rendered images can then be blended with corresponding projection images to generate composite images that are photorealistic.

First claim

What is claimed is:

1 . A computer-implemented method for rendering a head portrait, the method comprising:
rendering a first set of images, wherein each image included in the first set of images comprises one or more skin regions of a face in each image and excludes non-skin regions of the face in each image, the non-skin regions of the face including background pixels associated with non-facial content; determining a set of parameters based on one or more optimization operations, the first set of images, and a machine learning model; generating a second set of images based on the set of parameters and the machine learning model, wherein each image included in the second set of images comprises one or more skin regions associated with the face and one or more non-skin regions, wherein the set of parameters constrain the one or more skin regions of each image included in the second set of images to resemble one or more skin regions of a corresponding image included in the first set of images, and wherein the set of parameters constrain the one or more non-skin regions of each image included in the second set of images to correspond to non-skin regions of other images included in the second set of images; and blending each image included in the first set of images with a corresponding image included in the second set of images.

2 . The method of claim 1 , wherein determining the set of parameters comprises:
applying a plurality of weights to a plurality of basis vectors to determine an intermediate set of parameters; generating an intermediate set of images based on the intermediate set of parameters and the machine learning model; computing a rendering energy based on the intermediate set of parameters, the first set of images, and the intermediate set of images; computing an inpainting consistency energy based on the intermediate set of images; and updating the plurality of weights based on the rendering energy and the inpainting consistency energy.

3 . The method of claim 2 , wherein each weight included in the plurality of weights is associated with a corresponding segment of a basis vector included in the plurality of basis vectors.

4 . The method of claim 2 , wherein the rendering energy comprises a segmentation loss that penalizes misalignments between one or more portions of the one or more skin regions in the first set of images and corresponding one or more portions of one or more skin regions in the intermediate set of images.

5 . The method of claim 2 , wherein the inpainting consistency energy penalizes variations of one or more non-skin regions of the intermediate set of images from an average of the one or more non-skin regions of the intermediate set of images.

6 . The method of claim 2 , wherein the inpainting consistency energy penalizes differences between temporal neighbors in an ordering of the intermediate set of images.

7 . The method of claim 2 , further comprising selecting the plurality of basis vectors from a set of basis vectors.

8 . The method of claim 2 , further comprising sampling within a region around an origin of a latent space of the machine learning model to determine the plurality of basis vectors.

9 . The method of claim 1 , wherein each image included in the first set of images is blended with the corresponding image included in the second set of images based on a blurring of a corresponding mask that indicates one or more skin regions in the image included in the first set of images.

10 . The method of claim 1 , wherein the first set of images is rendered based on (i) geometry of the face that is at least one of captured, automatically generated, or manually created; and (ii) one or more associated appearance maps.

11 . One or more non-transitory computer-readable storage media including instructions that, when executed by at least one processor, cause the at least one processor to performing steps for rendering a head portrait, the steps comprising:
rendering a first set of images, wherein each image included in the first set of images comprises one or more skin regions of a face in each image and excludes non-skin regions of the face in each image, the non-skin regions of the face including background pixels associated with non-facial content; determining a set of parameters based on one or more optimization operations, the first set of images, and a machine learning model; generating a second set of images based on the set of parameters and the machine learning model, wherein each image included in the second set of images comprises one or more skin regions associated with the face and one or more non-skin regions, wherein the set of parameters constrain the one or more skin regions of each image included in the second set of images to resemble one or more skin regions of a corresponding image included in the first set of images, and wherein the set of parameters constrain the one or more non-skin regions of each image included in the second set of images to correspond to non-skin regions of other images included in the second set of images; and blending each image included in the first set of images with a corresponding image included in the second set of images.

12 . The one or more non-transitory computer-readable storage media of claim 11 , wherein determining the set of parameters comprises:
applying a plurality of weights to a plurality of basis vectors to determine an intermediate set of parameters; generating an intermediate set of images based on the intermediate set of parameters and the machine learning model; computing a rendering energy based on the intermediate set of parameters, the first set of images, and the intermediate set of images; computing an inpainting consistency energy based on the intermediate set of images; and updating the plurality of weights based on the rendering energy and the inpainting consistency energy.

13 . The one or more non-transitory computer-readable storage media of claim 12 , wherein each weight included in the plurality of weights is associated with a corresponding segment of a basis vector included in the plurality of basis vectors.

14 . The one or more non-transitory computer-readable storage media of claim 12 , wherein the rendering energy comprises a segmentation loss that penalizes misalignments between one or more portions of the one or more skin regions in the first set of images and corresponding one or more portions of one or more skin regions in the intermediate set of images.

15 . The one or more non-transitory computer-readable storage media of claim 12 , wherein the inpainting consistency energy penalizes at least one of (i) variations of one or more non-skin regions of the intermediate set of images from an average of the one or more non-skin regions of the intermediate set of images, or (ii) differences between temporal neighbors in an ordering of the intermediate set of images.

16 . The one or more non-transitory computer-readable storage media of claim 12 , wherein the second set of images is further generated based on a set of masks, each mask included in the set of masks indicating one or more skin regions in a corresponding image included in the first set of images.

17 . The one or more non-transitory computer-readable storage media of claim 12 , wherein the machine learning model comprises a style-based neural network.

18 . The one or more non-transitory computer-readable storage media of claim 11 , wherein each image included in the first set of images is blended with the corresponding image included

Assignees

• Disney Entpr Inc
• Eth Zuerich Eidgenoessische Technische Hochschule Zuerich

Inventors

• Bradley Derek Edward
• Chandran Prashanth
• Urnau Gotardo Paulo Fabiano
• Riviere Jeremy
• Winberg Sebastian Valentin
• Zoss Gaspard

Classifications

• G06V40/171

  Local features and components; Facial parts (eye characteristics G06V40/18); Occluding parts, e.g. glasses; Geometrical relationships · CPC title

• G06N3/08

  Learning methods · CPC title

• G06V40/165

  using facial parts and geometric relationships · CPC title

• G06T2207/30201

  Face · CPC title

• G06T15/503

  Blending, e.g. for anti-aliasing · CPC title