Volumetric avatars from a phone scan

Patent metadata

Field	Value
Publication number	US-2023245365-A1
Application number	US-202218074346-A
Country	US
Kind code	A1
Filing date	Dec 2, 2022
Priority date	Feb 1, 2022
Publication date	Aug 3, 2023
Grant date	—

Abstract

A method for generating a subject avatar using a mobile phone scan is provided. The method includes receiving, from a mobile device, multiple images of a first subject, extracting multiple image features from the images of the first subject based on a set of learnable weights, inferring a three-dimensional model of the first subject from the image features and an existing three-dimensional model of a second subject, animating the three-dimensional model of the first subject based on an immersive reality application running on a headset used by a viewer, and providing, to a display on the headset, an image of the three-dimensional model of the first subject. A system and a non-transitory, computer-readable medium storing instructions to perform the above method, are also provided.

First claim

What is claimed is:

1 . A computer-implemented method, comprising:
receiving, from a mobile device, multiple images of a first subject; extracting multiple image features from the images of the first subject based on a set of learnable weights; inferring a three-dimensional model of the first subject from the image features and an existing three-dimensional model of a second subject; animating the three-dimensional model of the first subject based on an immersive reality application running on a headset used by a viewer; and providing, to a display on the headset, an image of the three-dimensional model of the first subject.

2 . The computer-implemented method of claim 1 , wherein receiving multiple images of the first subject comprises receiving at least a neutral expression image of the first subject.

3 . The computer-implemented method of claim 1 , wherein receiving multiple images of the first subject comprises receiving at least an expressive image of the first subject.

4 . The computer-implemented method of claim 1 , wherein receiving multiple images of the first subject comprises receiving a sequence of images collected by scanning the mobile device in a selected direction over the first subject.

5 . The computer-implemented method of claim 1 , wherein inferring a three-dimensional model of the first subject comprises biasing the three-dimensional model of the first subject along a direction selected for collecting the images of the second subject.

6 . The computer-implemented method of claim 1 , wherein to form a three-dimensional model of the first subject comprises masking a gaze direction in the three-dimensional model of the second subject and inserting a gaze direction of the first subject.

7 . The computer-implemented method of claim 1 , wherein the image features comprise an identity feature of the first subject, and to form the three-dimensional model of the first subject comprises replacing an identity feature of the second subject with the identity feature of the second subject.

8 . The computer-implemented method of claim 1 , wherein the image features comprise an expression feature of the first subject, and to form the three-dimensional model of the first subject comprises matching the expression feature of the first subject in a latent expression database.

9 . The computer-implemented method of claim 1 , wherein animating the three-dimensional model of the first subject comprises projecting the image features along a direction between the three-dimensional model of the first subject and a selected observation point for the viewer.

10 . The computer-implemented method of claim 1 , wherein animating the three-dimensional model of the first subject comprises including an illumination source for the three-dimensional model of the first subject based on the existing three-dimensional model of the second subject.

11 . A system, comprising:
a memory storing multiple instructions; and one or more processors configured to execute the instructions to cause the system to perform operations, comprising: receive, from a mobile device, multiple images of a first subject; extract multiple image features from the images of the first subject based on a set of learnable weights; infer a three-dimensional model of the first subject from the image features and an existing three-dimensional model of a second subject; animate the three-dimensional model of the first subject based on an immersive application running on a headset used by a viewer; and provide, to a display on the headset, an image of the three-dimensional model of the first subject.

12 . The system of claim 11 , wherein to receive multiple images of the first subject the one or more processors are configured to receive at least a neutral expression image of the first subject.

13 . The system of claim 11 , wherein to receive multiple images of the first subject the one or more processors are configured to receive at least an expressive image of the first subject.

14 . The system of claim 11 , to receive multiple images of the first subject the one or more processors are configured to receive a sequence of images collected by scanning the mobile device in a selected direction over the first subject.

15 . The system of claim 11 , wherein to infer the three-dimensional model of the first subject the one or more processors are configured to bias the three-dimensional model of the first subject along a direction selected for collecting the images of the second subject.

16 . A computer-implemented method for training a model to provide a view of a subject to an auto stereoscopic display in a virtual reality headset, comprising:
collecting, from a face of multiple subjects, multiple images according to a capture script; updating an identity encoder and an expression encoder in a three-dimensional face model; generating, with the three-dimensional face model, a synthetic view of a user along a pre-selected direction corresponding to a view of the user; and training the three-dimensional face model based on a difference between an image of the user provided by a mobile device, and the synthetic view of the user.

17 . The computer-implemented method of claim 16 , wherein collecting multiple images according to a capture script comprises collecting each of the images with a pre-selected illumination configuration.

18 . The computer-implemented method of claim 16 , wherein collecting multiple images according to a capture script comprises collecting images with different expressions for each subject.

19 . The computer-implemented method of claim 16 , wherein training the three-dimensional face model comprises using a metric for a geometric artifact of the three-dimensional face model based on an image of the user.

20 . The computer-implemented method of claim 16 , wherein training the three-dimensional face model comprises using a metric for an identity artifact of the three-dimensional face model.

Assignees

• Meta Platforms Tech Llc

Inventors

• Cao Chen
• Anderson Stuart
• Kreuz Tomas Simon
• Kim Jin Kyu
• Schwartz Gabriel Bailowitz
• Zollhoefer Michael
• Saito Shunsuke
• Lombardi Stephen Anthony
• Wei Shih-En
• Belko Danielle
• Yu Shoou-I
• Sheikh Yaser
• Saragih Jason

Classifications

• G06T2219/2021

  Shape modification · CPC title

• G06T19/20

  Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts · CPC title

• G06T7/55

  from multiple images · CPC title

• G06T2207/30201

  Face · CPC title

• G06T13/40 Primary

  of characters, e.g. humans, animals or virtual beings · CPC title