Deep relightable appearance models for animatable face avatars

Patent metadata

Field	Value
Publication number	US-11715248-B2
Application number	US-202217580486-A
Country	US
Kind code	B2
Filing date	Jan 20, 2022
Priority date	Jan 26, 2021
Publication date	Aug 1, 2023
Grant date	Aug 1, 2023

Abstract

A method for providing a relightable avatar of a subject to a virtual reality application is provided. The method includes retrieving multiple images including multiple views of a subject and generating an expression-dependent texture map and a view-dependent texture map for the subject, based on the images. The method also includes generating, based on the expression-dependent texture map and the view-dependent texture map, a view of the subject illuminated by a light source selected from an environment in an immersive reality application, and providing the view of the subject to an immersive reality application running in a client device. A non-transitory, computer-readable medium storing instructions and a system that executes the instructions to perform the above method are also provided.

First claim

What is claimed is:

1. A computer-implemented method, comprising:
retrieving multiple images including multiple views of a subject;
generating an expression-dependent texture map and a view-dependent texture map for the subject, based on the images;
generating, based on the expression-dependent texture map and the view-dependent texture map, a view of the subject illuminated by a light source selected from an environment in an immersive reality application; and
providing the view of the subject to an immersive reality application running in a client device, wherein generating an expression-dependent texture map and a view-dependent texture map for the subject comprises interpolating a lighting configuration based on a first lighting configuration and a second lighting configuration available in the expression-dependent texture map and the view-dependent texture map.

2. The computer-implemented method of claim 1 , wherein the client device is a virtual reality headset and retrieving multiple images including multiple view of the subject comprises retrieving one or more frames from one or more headset mounted cameras facing a user of the virtual reality headset.

3. The computer-implemented method of claim 1 , wherein generating an expression-dependent texture map and a view-dependent texture map for the subject comprises selecting a lighting configuration for the immersive reality application.

4. The computer-implemented method of claim 1 , wherein generating an expression-dependent texture map and a view-dependent texture map for the subject comprises determining a lighting configuration based on an environment map including multiple lighting configurations in an environment for the subject in the immersive reality application.

5. The computer-implemented method of claim 1 , wherein generating an expression dependent texture map and a view-dependent texture map for the subject comprises determining a location of an environment of the subject in the immersive reality application, a subject orientation in the environment, and a view direction.

6. The computer-implemented method of claim 1 , wherein generating an expression-dependent texture map and a view-dependent texture map comprises retrieving a shadow map to encode a geometric association between a light source in the immersive reality application and the view-dependent texture map.

7. The computer-implemented method of claim 1 , wherein generating an expression-dependent texture map comprises linearly combining multiple expression dependent texture maps based on a lighting condition of the expression-dependent texture maps.

8. The computer-implemented method of claim 1 , wherein generating a view of the subject comprises identifying a clear shadow boundary from a self-occlusion from a portion of a face of the subject.

9. The computer-implemented method of claim 1 , further comprising providing a video of the subject based on animated views of the subject in the immersive reality application.

10. A system, comprising:
a memory storing multiple instructions; and
one or more processors configured to execute the instructions to cause the system to:
retrieve multiple images including multiple views of a subject;
generate an expression-dependent texture map and a view-dependent texture map for the subject, based on the images;
generate, based on the expression-dependent texture map and the view-dependent texture map, a view of the subject illuminated by a light source selected from an environment in an immersive reality application; and
provide the view of the subject to an immersive reality application running in a client device, wherein to generate an expression-dependent texture map and a view-dependent texture map for the subject the one or more processors execute instructions to interpolate a lighting configuration based on a first lighting configuration and a second lighting configuration available in the expression-dependent texture map and the view-dependent texture map.

11. The system of claim 10 , wherein the client device is a virtual reality headset and to retrieve multiple images including multiple view of the subject the one or more processors execute instructions to retrieve one or more frames from one or more headset mounted cameras facing a user of the virtual reality headset.

12. The system of claim 10 , wherein to generate an expression-dependent texture map and a view-dependent texture map for the subject the one or more processors execute instructions to select a lighting configuration for the immersive reality application.

13. The system of claim 10 , wherein to generate an expression-dependent texture map and a view-dependent texture map for the subject the one or more processors execute instructions to determine a lighting configuration based on an environment map including multiple lighting configurations in an environment for the subject in the immersive reality application.

14. A computer-implemented method for training a model to generate a relightable, three-dimensional representation of a subject, comprising:
retrieving multiple images including multiple views of a subject under multiple space-multiplexed and time-multiplexed illumination patterns;
generating, with a relightable appearance model, an expression-dependent texture map and a view-dependent texture map for the subject, based on the images;
generating, based on the expression-dependent texture map and the view-dependent texture map, a synthetic view of the subject illuminated by each of the space-multiplexed and time-multiplexed illumination patterns;
determining a loss value indicative of a difference between the synthetic view of the subject and at least one of the images including multiple views of the subject;
updating the relightable appearance model based on the loss value; and
storing the relightable appearance model in a memory circuit.

15. The computer-implemented method of claim 14 , wherein retrieving multiple images including multiple views of a subject comprises configuring an array of light emitters in a one light at a time illumination pattern.

16. The computer-implemented method of claim 14 , wherein retrieving multiple images including multiple views of a subject comprises selecting a time-multiplexed pattern of lighting configurations including complementary lighting configurations, a fully-lit lighting configuration, and a single light lighting configuration.

17. The computer-implemented method of claim 14 , wherein generating an expression-dependent texture map and a view-dependent texture map for the subject comprises generating pixelated frames having a color value and an opacity value for each of multiple expressions and multiple view directions of the subject from the images.

18. The computer-implemented method of claim 14 , wherein generating an expression-dependent texture map and a view-dependent texture map for the subject comprises generating a shadow map that associates each light in an environment of the subject with a view direction and an occlusion along the view direction.

19. The computer-implemented method of claim 14 , further comprising providing the relightable, three-dimensional representation of the subject to a client device for an immersive reality application.

Assignees

• Meta Platforms Tech Llc

Inventors

• Saragih Jason
• Lombardi Stephen Anthony
• Saito Shunsuke
• Simon Kreuz Tomas
• Wei Shih-En
• Mcphail Kevyn Alex Anthony
• Sheikh Yaser
• Bi sai

Classifications

• G06T11/10

  Texturing; Colouring; Generation of textures or colours (retouching, inpainting or scratch removal G06T5/77) · CPC title

• G06T13/40 Primary

  of characters, e.g. humans, animals or virtual beings · CPC title

• G06T11/001

  Physics · mapped topic

• G06T15/04

  Texture mapping · CPC title

• G06T15/506 Primary

  Illumination models · CPC title