Avatar personalization using image generation

Patent metadata

Field	Value
Publication number	US-2024257470-A1
Application number	US-202318458731-A
Country	US
Kind code	A1
Filing date	Aug 30, 2023
Priority date	Jan 30, 2023
Publication date	Aug 1, 2024
Grant date	—

Abstract

A method and system for personalized avatar generation. The method includes receiving a text prompt and generating a first pose based on the text prompt using a first model. The method also includes re-targeting the first pose to a target avatar body. The method also includes identifying a predefined avatar configuration corresponding to a user based on a profile of the user. The method also includes converting the target avatar body by applying the predefined avatar configuration to the target avatar body. The method also includes rendering an avatar, using a second model, based on the target avatar body with the predefined avatar configuration, wherein the avatar is in the first pose.

First claim

What is claimed is:

1 . A computer-implemented method, performed by at least one processor, for personalized avatar generation, the method comprising:
receiving a text prompt; generating a first pose based on the text prompt using a first model; re-targeting the first pose to a target avatar body; identifying a predefined avatar configuration corresponding to a user based on a profile of the user; converting the target avatar body by applying the predefined avatar configuration to the target avatar body; and rendering an avatar, using a second model, based on the target avatar body with the predefined avatar configuration, wherein the avatar is in the first pose.

2 . The computer-implemented method of claim 1 , wherein the text prompt describes a scene and interactions of the avatar within the scene.

3 . The computer-implemented method of claim 1 , wherein the first pose is a 3D pose generated based on human joint and limb orientation and positioning parameters.

4 . The computer-implemented method of claim 1 , wherein the first pose is a human body pose represented by SMPL parameters.

5 . The computer-implemented method of claim 1 , the generating the first pose further comprising:
extracting first text embeddings from the text prompt; mapping the first text embeddings to second text embeddings retrieved from a dataset; selecting one or more second poses corresponding to the second text embeddings; and determining the first pose based on the one or more second poses.

6 . The computer-implemented method of claim 1 , further comprising:
training the first model on a dataset including body poses and corresponding text descriptions, the training comprising:
identifying humans in images of the dataset,
segmenting the humans from the images, and
extracting 3D Skinned Multi-Person Linear Model (SMPL) annotations from segmented humans from the images.

7 . The computer-implemented method of claim 1 , wherein the target avatar body is a gray avatar-human body representation.

8 . The computer-implemented method of claim 1 , wherein the re-targeting comprises matching corresponding joints, in position and orientation, from the first pose and the target avatar body.

9 . The computer-implemented method of claim 1 , further comprising generating an image, using the second model, of a scene in a virtual environment including the avatar interacting with objects in the scene.

10 . The computer-implemented method of claim 9 , wherein the second model performs conditional stable diffusion inpainting to generate the image by outpainting from the avatar to fill in the scene and objects in the scene, and the second model is conditioned on at least the avatar and the text prompt.

11 . A system for personalized avatar generation, the system comprising:
one or more processors; and a memory storing instructions which, when executed by the one or more processors, cause the system to:
receive a text prompt describing a scene and interactions of the avatar within the scene;
generate a first pose based on the text prompt using a first model;
re-target the first pose to a target avatar body;
identify a predefined avatar configuration corresponding to a user based on a profile of the user;
convert the target avatar body by applying the predefined avatar configuration to the target avatar body; and
render an avatar, using a second model, based on the target avatar body with the predefined avatar configuration, wherein the avatar is in the first pose.

12 . The system of claim 11 , wherein the first pose is a 3D pose generated based on human joint and limb orientation and positioning parameters.

13 . The system of claim 11 , wherein the first pose is a human body pose represented by SMPL parameters.

14 . The system of claim 11 , wherein the one or more processors further execute instructions to:
extract first text embeddings from the text prompt; map the first text embeddings to second text embeddings retrieved from a dataset; select one or more second poses corresponding to the second text embeddings; and determine the first pose based on the one or more second poses.

15 . The system of claim 11 , wherein the one or more processors further execute instructions to train the first model on a dataset including body poses and corresponding text descriptions, the instructions causing the system to:
identify humans in images of the dataset; segment the humans from the images; and extract 3D Skinned Multi-Person Linear Model (SMPL) annotations from segmented humans from the images.

16 . The system of claim 11 , wherein the target avatar body is a gray avatar-human body representation.

17 . The system of claim 11 , wherein the one or more processors further execute instructions to match corresponding joints, in position and orientation, from the first pose and the target avatar body.

18 . The system of claim 11 , wherein the one or more processors further execute instructions to:
generate an image, using the second model, of a scene in a virtual environment including the avatar interacting with objects in the scene.

19 . The system of claim 11 , wherein the second model performs conditional stable diffusion inpainting to generate an image by outpainting from the avatar to fill in the scene and objects in the scene, and the second model is conditioned on at least the avatar and the text prompt.

20 . A non-transient computer-readable storage medium having instructions embodied thereon, the instructions being executable by one or more processors to perform a method for personalized avatar generation and cause the one or more processors to:
receive a text input describing an avatar in a scene; generating a body pose based on the text prompt; re-targeting the body pose to a target avatar body; generating a personalized avatar based on a predefined avatar configuration being applied to the target avatar body; and generating an image of the avatar in the scene based on the personalized avatar, the image including the avatar being in the body pose with the predefined avatar configuration.

Assignees

• Meta Platforms Tech Llc

Inventors

• Gupta Sonal
• Azadi Samaneh
• Shah Mian Akbar
• Hayes Thomas Falstad
• Parikh Devi Niru

Classifications

• A63F13/67

  adaptively or by learning from player actions, e.g. skill level adjustment or by storing successful combat sequences for re-use · CPC title

• A63F13/63

  by the player, e.g. authoring using a level editor · CPC title

• A63F13/55

  Controlling game characters or game objects based on the game progress · CPC title

• A63F13/52

  involving aspects of the displayed game scene · CPC title

• G06T13/40

  of characters, e.g. humans, animals or virtual beings · CPC title