Applying animated 3D avatar in AR experiences

Patent metadata

Field	Value
Publication number	US-12307564-B2
Application number	US-202217888786-A
Country	US
Kind code	B2
Filing date	Aug 16, 2022
Priority date	Jul 7, 2022
Publication date	May 20, 2025
Grant date	May 20, 2025

Abstract

Aspects of the present disclosure involve a system for providing virtual experiences. The system accesses, by a messaging application, an image depicting a person. The system generates, by the messaging application, a three-dimensional (3D) avatar based on the person depicted in the image. The system receives input that selects a pose for the 3D avatar and one or more fashion items to be worn by the 3D avatar and places, by the messaging application, the 3D avatar in the selected pose and wearing the one or more fashion items in an augmented reality (AR) experience.

First claim

What is claimed is:

1. A method comprising:
accessing, by an application, an image depicting a person;
generating, by the application, a three-dimensional (3D) avatar based on the person depicted in the image in a real-world environment;
receiving input that selects one or more fashion items to be worn by the 3D avatar;
placing, by the messaging application, the 3D avatar wearing the one or more fashion items in an augmented reality (AR) experience;
presenting the 3D avatar in a virtual environment defined by the AR experience different from the real-world environment in which the person is depicted in the image used to generate the 3D avatar;
identifying one or more animation parameters associated with the one or more fashion items, the one or more animation parameters specifying an initial position to display the 3D avatar, a direction along which the 3D avatar is animated as moving, a set of poses that the 3D avatar performs while moving along the direction, and a final position to display the 3D avatar; and
in response to identifying the one or more animation parameters, using the one or more animation parameters to continuously modify a pose or movement of the 3D avatar to perform the set of poses along the direction specified by the one or more animation parameters in the virtual environment in which the 3D avatar is presented.

2. The method of claim 1 , wherein the 3D avatar is presented in a new pose that is different from a pose of the person depicted in the image, further comprising:
receiving a user selection of the AR experience from a list of AR experiences.

3. The method of claim 1 , further comprising automatically selecting the AR experience from a list of AR experiences, wherein the 3D avatar is adjusted to be in any position and to wear any selected fashion item in an automatically fitted manner.

4. The method of claim 1 , further comprising:
activating a camera of a client device to capture a real-time video feed depicting a real-world environment;
adding one or more AR elements to the real-world environment depicted in the real-time video feed; and
displaying a modified real-time video feed comprising the 3D avatar, the one or more AR elements, and the depiction of the real-world environment.

5. The method of claim 4 , wherein the one or more AR elements correspond to a runway that is part of a modeling show.

6. The method of claim 1 , further comprising:
obtaining one or more animation parameters associated with the AR experience; and
automatically animating the 3D avatar using the one or more animation parameters for display together with one or more AR elements in a real-time video feed.

7. The method of claim 1 , further comprising selecting a 3D position in which to place the 3D avatar.

8. The method of claim 1 , further comprising rigging the 3D avatar based on features of the image.

9. The method of claim 1 , further comprising:
receiving a swipe gesture from a user; and
in response to receiving the swipe gesture, modifying the one or more fashion items worn by the 3D avatar.

10. The method of claim 1 , wherein the one or more fashion items include at least one of shirt, pants, skirt, dress, jewelry, purse, furniture item, household item, eyewear, eyeglasses, AR logos, AR emblems, purse, pants, shorts, skirts, jackets, t-shirts, blouses, glasses, jewelry, earrings, bunny ears, a hat, ear muffs, or facial makeup.

11. The method of claim 1 , further comprising:
obtaining one or more animation parameters associated with the one or more fashion items; and
automatically animating the 3D avatar using the one or more animation parameters, associated with the one or more fashion items, for display together with one or more AR elements in a real-time video feed.

12. The method of claim 1 , further comprising training a neural network to generate the 3D avatar, the neural network configured to establish a relationship between points of an image and points in a volume corresponding to a given 3D avatar.

13. The method of claim 12 , further comprising:
generating a segmentation of the person depicted in the image;
applying the neural network to generate a volume reconstruction of the person using the segmentation and to output a texture of the person; and
fitting the volume reconstruction and the texture to a 3D avatar template.

14. The method of claim 13 , further comprising:
unposing the 3D avatar template fitted with the volume reconstruction and the texture; and
applying an animation to the unposed 3D avatar.

15. The method of claim 12 , further comprising performing training operations comprising:
receiving training data comprising a plurality of training images depicting a training person and ground-truth meshes of the training person;
applying the neural network to a first training image of the plurality of training images to estimate a volume reconstruction of the training person depicted in the first training image, the volume reconstruction indicating whether each point of the first training image is inside or outside of a given mesh;
obtaining the ground-truth mesh corresponding to the first training image;
comparing the volume reconstruction to the ground-truth mesh corresponding to the first training image to compute a deviation; and
updating parameters of the neural network based on the computed deviation.

16. The method of claim 1 , further comprising automatically aligning the one or more fashion items on the 3D avatar.

17. A system comprising:
at least one processor configured to perform operations comprising:
accessing, by an application, an image depicting a person;
generating, by the application, a three-dimensional (3D) avatar based on the person depicted in the image in a real-world environment;
receiving input that selects one or more fashion items to be worn by the 3D avatar;
placing, by the application, the 3D avatar wearing the one or more fashion items in an augmented reality (AR) experience;
presenting the 3D avatar in a virtual environment defined by the AR experience different from the real-world environment in which the person is depicted in the image used to generate the 3D avatar;
identifying one or more animation parameters associated with the one or more fashion items, the one or more animation parameters specifying an initial position to display the 3D avatar, a direction along which the 3D avatar is animated as moving, a set of poses that the 3D avatar performs while moving along the direction, and a final position to display the 3D avatar; and
in response to identifying the one or more animation parameters, using the one or more animation parameters to continuously modify a pose or movement of the 3D avatar to perform the set of poses along the direction specified by the one or more animation parameters in the virtual environment in which the 3D avatar is presented.

18. The system of claim 17 , the operations further comprising:
receiving a user selection of the AR experience from a list of AR experiences, wherein different fashion items cause the 3D avatar to be animated in different ways.

19. The system of claim 17 , the operations further comprising automatically selecting the AR experience from a list of AR experiences.

20. A non-transitory machine-readable storage medium that includes instructions that, when executed by one or more processors of a machine, cause the machine to perform operations comprising:
accessing, by an application, an image depicting a person;
generating, by the application, a three

Assignees

• Snap Inc

Inventors

• Assouline Avihay
• Berger Itamar
• Guler Riza Alp
• Kakolyris Antonios
• Lu Frank
• Wang Haoyang
• Zohar Matan

Classifications

• G06T19/006

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• H04L51/10 Primary

  Multimedia information · CPC title

• H04W4/21

  for social networking applications · CPC title

• G06T13/40 Primary

  of characters, e.g. humans, animals or virtual beings · CPC title