Inferred shading

Patent metadata

Field	Value
Publication number	US-11967018-B2
Application number	US-202017129158-A
Country	US
Kind code	B2
Filing date	Dec 21, 2020
Priority date	Dec 20, 2019
Publication date	Apr 23, 2024
Grant date	Apr 23, 2024

Abstract

Rendering an avatar in a selected environment may include determining as inputs into an inferred shading network, an expression geometry to be represented by an avatar, head pose, and camera angle, along with a lighting representation for the selected environment. The inferred shading network may then generate a texture of a face to be utilized in rendering the avatar. The lighting representation may be obtained as lighting latent variables which are obtained from an environment autoencoder trained on environment images with various lighting conditions.

First claim

The invention claimed is:

1. A non-transitory computer readable medium comprising computer readable instructions executable by one or more processors to:
receive lighting latents for an environment in which an avatar of a subject is to be displayed
wherein the subject is in a separate environment having different lighting than the environment in which the avatar is to be displayed;
determine mesh latents for the subject;
apply the lighting latents and the mesh latents to an inferred shading network comprising a lighting encoder and a texture decoder,
wherein the lighting encoder generates a set of texture latents based on the lighting latents and mesh latents, and
wherein the texture decoder generates a texture map representing a texture of a face of the subject in the lighting of the environment based on the set of texture latents provided by the lighting encoder; and
render an avatar of the subject in a view of the environment based on the texture map and a geometric representation of the subject.

2. The non-transitory computer readable medium of claim 1 , wherein the lighting encoder translates the lighting latents to the set of texture latents, and wherein the texture decoder generates the texture map from the set of texture latents.

3. The non-transitory computer readable medium of claim 1 further comprising computer readable code to:
obtain a plurality of environment map images; and
train an environmental autoencoder based on the environment map images,
wherein the lighting latents are representative of a lighting of an image of an input scene into the trained environmental autoencoder.

4. The non-transitory computer readable medium of claim 1 , wherein the computer readable code for rendering the avatar further comprises computer readable code to:
obtain a mesh representation of the subject; and
apply the texture map to the mesh representation.

5. The non-transitory computer readable medium of claim 1 , wherein the lighting latents comprises a compressed representation of color in the environment.

6. The non-transitory computer readable medium of claim 1 , wherein the lighting latents comprises at least one selected from a group consisting of spherical harmonic coefficients, spherical gaussians, and spherical wavelets for the environment.

7. A system for generating an avatar, comprising:
one or more processors; and
a non-transitory computer readable medium comprising computer readable instructions executable by the one or more processors to:
receive lighting latents for an environment in which an avatar of a subject is to be displayed, wherein the subject is in a separate environment having different lighting than the environment in which the avatar is to be displayed;
determine mesh latents for the subject;
apply the lighting latents and the mesh latents to an inferred shading network comprising a lighting encoder and a texture decoder,
wherein the lighting encoder generates a set of texture latents based on the lighting latents and mesh latents, and
wherein the texture decoder generates a texture map representing a texture of a face of the subject in the lighting of the environment based on the set of texture latents provided by the lighting encoder; and
render an avatar of the subject in a view of the environment based on the texture map and a geometric representation of the subject.

8. The system of claim 7 , wherein the lighting encoder translates the lighting latents to the set of texture latents, and wherein the texture decoder generates the texture map from the set of texture latents.

9. The system of claim 7 , further comprising computer readable code to:
obtain a plurality of environment images; and
train an environmental autoencoder based on the environment images,
wherein the lighting latents are representative of a lighting of an image of an input scene into the trained environmental autoencoder.

10. The system of claim 9 , wherein the lighting latents comprises a compressed representation of color in the environment.

11. The system of claim 7 , wherein the computer readable code for rendering the avatar further comprises computer readable code to:
obtain a mesh representation of the subject; and
apply the texture map to the mesh representation.

12. The system of claim 7 , wherein the lighting latents comprises spherical harmonic coefficients for the environment.

13. A method for generating an avatar, comprising:
receiving lighting latents for an environment in which an avatar of a subject is to be displayed, wherein the subject is in a separate environment having different lighting than the environment in which the avatar is to be displayed;
determining mesh latents for the subject;
applying the lighting latents and the mesh latents to an inferred shading network comprising a lighting encoder and a texture decoder,
wherein the lighting encoder generates a set of texture latents based on the lighting latents and mesh latents, and
wherein the texture decoder generates a texture map representing a texture of a face of the subject in the lighting of the environment based on the set of texture latents provided by the lighting encoder; and
rendering an avatar of the subject in a view of the environment based on the texture map and a geometric representation of the subject.

14. The method of claim 13 , further comprising:
obtaining a plurality of environment images; and
training an environmental autoencoder based on the environment images,
wherein the lighting latents are representative of a lighting of an image of an input scene into the trained environmental autoencoder.

15. The method of claim 13 , wherein rendering the avatar further comprises:
obtaining a mesh representation of the subject; and
applying the texture map to the mesh representation.

16. The method of claim 13 , wherein the lighting latents comprises a compressed representation color in the environment.

17. The method of claim 13 , wherein the lighting latents comprises spherical harmonic coefficients or spherical Gaussians for the environment.

Assignees

• Mason Andrew P
• Soares Olivier
• Duiker Haarm Pieter
• Mccarten John S
• Apple Inc

Inventors

• Mason Andrew P
• Soares Olivier
• Duiker Haarm-Pieter
• Mccarten John S

Classifications

• G06T15/80 Primary

  Shading · CPC title

• G06T7/507

  from shading (G06T7/586 takes precedence) · CPC title

• G06T7/529

  from texture · CPC title

• G06T13/40

  of characters, e.g. humans, animals or virtual beings · CPC title

• G06T15/04 Primary

  Texture mapping · CPC title