Generating avatar fashion items

Patent metadata

Field	Value
Publication number	US-2025342621-A1
Application number	US-202418656196-A
Country	US
Kind code	A1
Filing date	May 6, 2024
Priority date	May 6, 2024
Publication date	Nov 6, 2025
Grant date	—

Abstract

Methods and systems are disclosed for using generative machine learning models to generate fashion items for avatars. The methods and systems receive an image depicting a set of fashion items and identify a set of three-dimensional (3D) avatar fashion item assets corresponding to the set of fashion items depicted in the image. The methods and systems replace textures of the set of 3D avatar fashion item assets with target textures generated using the set of fashion items depicted in the image to generate a set of target avatar fashion item assets and generate an avatar using the set of target avatar fashion item assets.

First claim

What is claimed is:

1 . A method comprising:
receiving, by one or more processors, an image depicting a set of fashion items; identifying a set of three-dimensional (3D) avatar fashion item assets corresponding to the set of fashion items depicted in the image; replacing textures of the set of 3D avatar fashion item assets with target textures generated using the set of fashion items depicted in the image to generate a set of target avatar fashion item assets; and generating an avatar using the set of target avatar fashion item assets.

2 . The method of claim 1 , wherein the set of 3D avatar fashion item assets comprises 3D mesh primitive avatar fashion item assets.

3 . The method of claim 1 , further comprising:
processing the image using a segmentation model to detect and segment each fashion item in the set of fashion items depicted in the image.

4 . The method of claim 3 , further comprising:
searching a database of 3D avatar fashion item assets for 3D avatar fashion item assets that match a type of each fashion item that has been detected and segmented to output the set of 3D avatar fashion item assets.

5 . The method of claim 1 , further comprising:
processing, using a generative machine learning model, the received image depicting the set of fashion items together with the set of 3D avatar fashion item assets to generate the target textures.

6 . The method of claim 5 , further comprising:
generating a prompt with instructions for the generative machine learning model to process the image depicting the set of fashion items and a mesh associated with the set of 3D avatar fashion item assets and to generate realistic style textures of a front portion and back portion of each fashion item in the set of 3D avatar fashion item assets to match textures of the set of fashion items.

7 . The method of claim 6 , further comprising:
processing, by a machine learning model, the realistic style textures of the front portion and back portion of each fashion item in the set of 3D avatar fashion item assets to estimate the target textures comprising avatar style textures.

8 . The method of claim 7 , wherein the machine learning model is trained by performing training operations comprising:
accessing training data comprising a first training image depicting a training fashion item in a realistic style texture and a second ground truth image depicting the training fashion item in an avatar style texture; analyzing, using the machine learning model, the first training image to estimate an avatar style texture for the training fashion item; computing a loss based on a deviation between the estimated avatar style texture for the training fashion item and the ground truth image depicting the training fashion item in the avatar style texture; and updating one or more parameters of the machine learning model based on the computed loss.

9 . The method of claim 8 , wherein the avatar style textures appear flatter than the realistic style textures, wherein the avatar style textures have less light than the realistic style textures, wherein the avatar style textures have fewer wrinkles than the realistic style textures, and wherein the avatar style textures have fewer shadows than the realistic style textures.

10 . The method of claim 1 , wherein the set of fashion items comprises a portion of an upper body fashion item, further comprising:
determining that the image partially depicts a front portion of the upper body fashion item; identifying an upper body 3D avatar fashion item asset as part of the set of 3D avatar fashion item assets; and generating the target textures to include an entire front portion of the upper body 3D avatar fashion item asset corresponding to the partially depicted front portion of the upper body fashion item and a back portion of the upper body 3D avatar fashion item asset corresponding to an artificial rendering of a back portion of the upper body fashion item.

11 . The method of claim 10 , further comprising:
determining that the set of fashion items excludes a lower body fashion item; in response to determining that the set of fashion items excludes the lower body fashion item, selecting a lower body 3D avatar fashion item asset that matches visual attributes of the upper body fashion item depicted in the image.

12 . The method of claim 11 , further comprising:
generating the target textures to include a front portion of the lower body 3D avatar fashion item asset based on the visual attributes of the upper body fashion item and a back portion of the lower body 3D avatar fashion item asset based on the visual attributes of the upper body fashion item.

13 . The method of claim 1 , wherein the avatar is generated to have a full body outfit that covers a top portion of the avatar above a waist of the avatar and a lower portion of the avatar below the waist.

14 . The method of claim 1 , wherein the image is captured by a camera of a user system.

15 . The method of claim 1 , further comprising:
generating a prompt with instructions to generate an artificial image depicting one or more artificial fashion items; and processing the prompt, by a generative machine learning model, to generate the artificial image depicting the one or more artificial fashion items, wherein the artificial image is received as the image depicting the set of fashion items.

16 . The method of claim 1 , further comprising:
overlaying the avatar on a video depicting a real-world environment.

17 . The method of claim 16 , further comprising:
animating the avatar that has been generated using the set of target avatar fashion item assets.

18 . A system comprising:
at least one processor; and at least one memory component having instructions stored thereon that, when executed by the at least one processor, cause the at least one processor to perform operations comprising: receiving an image depicting a set of fashion items; identifying a set of three-dimensional (3D) avatar fashion item assets corresponding to the set of fashion items depicted in the image; replacing textures of the set of 3D avatar fashion item assets with target textures generated using the set of fashion items depicted in the image to generate a set of target avatar fashion item assets; and generating an avatar using the set of target avatar fashion item assets.

19 . The system of claim 18 , the operations comprising:
overlaying the avatar on a video depicting a real-world environment.

20 . A non-transitory computer-readable storage medium having stored thereon instructions that, when executed by at least one processor, cause the at least one processor to perform operations comprising:
receiving an image depicting a set of fashion items; identifying a set of three-dimensional (3D) avatar fashion item assets corresponding to the set of fashion items depicted in the image; replacing textures of the set of 3D avatar fashion item assets with target textures generated using the set of fashion items depicted in the image to generate a set of target avatar fashion item assets; and generating an avatar using the set of target avatar fashion item assets.

Assignees

• Snap Inc

Inventors

• Assouline Avihay
• Berger Itamar
• Dudovitch Gal
• Harel Peleg

Classifications

• G06T11/10 Primary

  Texturing; Colouring; Generation of textures or colours (retouching, inpainting or scratch removal G06T5/77) · CPC title

• G06T15/04

  Texture mapping · CPC title

• G06N20/00

  Machine learning · CPC title

• G06T7/10

  Segmentation; Edge detection (motion-based segmentation G06T7/215) · CPC title

• G06T2207/10016

  Video; Image sequence · CPC title