Controllable dynamic appearance for neural 3D portraits

Patent metadata

Field	Value
Publication number	US-12437492-B2
Application number	US-202318132272-A
Country	US
Kind code	B2
Filing date	Apr 7, 2023
Priority date	Apr 7, 2023
Publication date	Oct 7, 2025
Grant date	Oct 7, 2025

Abstract

Certain aspects and features of this disclosure relate to providing a controllable, dynamic appearance for neural 3D portraits. For example, a method involves projecting a color at points in a digital video portrait based on location, surface normal, and viewing direction for each respective point in a canonical space. The method also involves projecting, using the color, dynamic face normals for the points as changing according to an articulated head pose and facial expression in the digital video portrait. The method further involves disentangling, based on the dynamic face normals, a facial appearance in the digital video portrait into intrinsic components in the canonical space. The method additionally involves storing and/or rendering at least a portion of a head pose as a controllable, neural 3D portrait based on the digital video portrait using the intrinsic components.

First claim

What is claimed is:

1. A method comprising:
projecting a color at a plurality of points in a digital video portrait based on a location, a surface normal, and a viewing direction for each respective point in a canonical space;
defining a neural radiance field as a continuous function that outputs color and density for the digital video portrait regardless of lighting;
providing, using the neural radiance field for the digital video portrait, a guided deformation field;
projecting, using the color and based on the guided deformation field, a dynamic face normal relative to a surface at each respective point of the plurality of points as changed relative to the surface normal based on an articulated head pose and facial expression in the digital video portrait;
disentangling, based on the dynamic face normal for each respective point of the plurality of points, a facial appearance in the digital video portrait into a plurality of intrinsic components in the canonical space; and
rendering at least a portion of a head pose as a controllable, neural three-dimensional portrait based on the digital video portrait using the plurality of intrinsic components.

2. The method of claim 1 , further comprising:
photographically capturing the digital video portrait; and
projecting a photometrically consistent albedo using the digital video portrait to project the color at the plurality of points.

3. The method of claim 1 , further comprising defining the canonical space with respect to a photometrically consistent albedo for the digital video portrait.

4. The method of claim 1 , further comprising:
determining a photometrically consistent albedo, a shading, and a specularity at each respective point to project the color, wherein the photometrically consistent albedo represents a constant color, defined to be the same despite variations in ambient light under which the digital video portrait is captured for training; and
defining the canonical space based on the photometrically consistent albedo, the shading, and the specularity.

5. The method of claim 1 , further comprising deforming each of the plurality of points using the guided deformation field to provide the dynamic face normal.

6. The method of claim 5 , further comprising:
training parameters of the neural radiance field to minimize a difference between an expected color and ground truth for each of the plurality of points;
training a deformation field using coarse-to-fine and vertex deformation regularization; and
extending the neural radiance field using the deformation field and the parameters as trained to produce the guided deformation field.

7. The method of claim 1 , further comprising:
producing a three-dimensional morphable model of the digital video portrait; and
accessing the three-dimensional morphable model for each of the plurality of points to provide the guided deformation field for each respective point of the plurality of points.

8. A system comprising:
a memory component; and
a processing device coupled to the memory component, the processing device to perform operations comprising:
projecting a color at a plurality of points in a digital video portrait based on a location, a surface normal, and a viewing direction for each respective point in a canonical space;
defining a neural radiance field as a continuous function that outputs color and density for the digital video portrait regardless of lighting;
providing, using a neural radiance field for the digital video portrait, a guided deformation field;
projecting, using the color and the guided deformation field, a dynamic face normal relative to a surface at each respective point of the plurality of points as changed relative to the surface normal based on an articulated head pose and facial expression in the digital video portrait;
disentangling, based on the dynamic face normal for each respective point of the plurality of points, a facial appearance in the digital video portrait into a plurality of intrinsic components in the canonical space; and
rendering or storing at least a portion of a head pose as a controllable, neural three-dimensional portrait based on the digital video portrait using the plurality of intrinsic components.

9. The system of claim 8 , wherein the operations further comprise:
photographically capturing the digital video portrait; and
projecting a photometrically consistent albedo using the digital video portrait to project the color at the plurality of points.

10. The system of claim 8 , wherein the operations further comprise defining the canonical space with respect to a photometrically consistent albedo for the digital video portrait.

11. The system of claim 8 , wherein the operations further comprise:
determining a photometrically consistent albedo, a shading, and a specularity at each respective point to project the color, wherein the photometrically consistent albedo represents a constant color, defined to be the same despite variations in ambient light under which the digital video portrait is captured for training; and
defining the canonical space based on the photometrically consistent albedo, the shading, and the specularity.

12. The system of claim 8 , wherein the operations further comprise deforming each of the plurality of points using the guided deformation field to provide the dynamic face normal.

13. The system of claim 12 , wherein the operations further comprise:
training parameters of the neural radiance field to minimize a difference between an expected color and ground truth for each of the plurality of points;
training a deformation field using coarse-to-fine and vertex deformation regularization; and
extending the neural radiance field using the deformation field and the parameters as trained to produce the guided deformation field.

14. The system of claim 8 , wherein the operations further comprise:
producing a three-dimensional morphable model of the digital video portrait; and
accessing the three-dimensional morphable model for each of the plurality of points to provide the guided deformation field for each respective point of the plurality of points.

15. A non-transitory computer-readable medium storing executable instructions, which when executed by a processing device, cause the processing device to perform operations comprising:
projecting a color at a plurality of points in a digital video portrait based on location, surface normal, and viewing direction for each respective point in a canonical space;
defining a neural radiance field as a continuous function that outputs color and density for the digital video portrait regardless of lighting;
providing, using a neural radiance field for the digital video portrait, a guided deformation field;
a step for producing, using the color at the plurality of points and based on the guided deformation field, intrinsic components of a controllable, neural three-dimensional portrait in the canonical space based on a facial appearance in the digital video portrait; and
rendering at least a portion of a head pose using the controllable, neural three-dimensional portrait using the intrinsic components.

16. The non-transitory computer-readable medium of claim 15 , wherein the executable instructions further cause the processing device to perform operations comprising:
photographically capturing the digital video portrait; and
projecting a photometrically consistent albedo using the digital video portrait to project the color at the plurality of points.

17. The non-transitory computer-readable medium of claim 15 , wh

Assignees

• Adobe Inc

Inventors

• Shu Zhixin
• Xu Zexiang
• Athar Shahrukh
• Bi sai
• Sunkavalli Kalyan
• Luan fujun

Classifications

• G06T2219/2021

  Shape modification · CPC title

• G06T2219/2012

  Colour editing, changing, or manipulating; Use of colour codes · CPC title

• G06T2210/44

  Morphing · CPC title

• G06T17/20

  Finite element generation, e.g. wire-frame surface description, {tesselation} · CPC title

• G06T15/80

  Shading · CPC title