Electronic apparatus and method for controlling thereof

Patent metadata

Field	Value
Publication number	US-12444124-B2
Application number	US-202318208976-A
Country	US
Kind code	B2
Filing date	Jun 13, 2023
Priority date	Oct 29, 2021
Publication date	Oct 14, 2025
Grant date	Oct 14, 2025

Abstract

Provide is an electronic apparatus that includes a communication interface, a memory storing at least one instruction, and a processor configured to execute the at least one instruction to: obtain an image including a person object, obtain a 3D shape model corresponding to the person object included in the image, map a texture of the image to the 3D shape model based on identification information for each area of the 3D shape model, and generate a 3D avatar corresponding to the person object based on the 3D shape model to which the texture of the image is mapped.

First claim

What is claimed is:

1. An electronic apparatus comprising:
a communication interface including communication circuitry;
a memory storing at least one instruction; and
a processor configured to execute the at least one instruction to:
obtain an image including a person object;
obtain a face image including a face of the person object;
obtain a 3D face model of the person object based on the face image;
obtain a three-dimensional (3D) shape model corresponding to the person object included in the image by mapping a texture of the image to a first area of the 3D shape model to generate a first 3D shape model and mapping the texture of the image to a second area of the 3D shape model based on identification information, the second area being different than the first area to generate a second 3D shape model; and
generate a 3D avatar corresponding to the person object based on the 3D shape model to which the texture of the image is mapped.

2. The electronic apparatus of claim 1 , wherein the processor is further configured to execute the at least one instruction to obtain the identification information for each area of the 3D shape model by performing image segmentation with respect to the first 3D shape model.

3. The electronic apparatus of claim 1 , wherein the processor is further configured to execute the at least one instruction to:
obtain information regarding an area in which the person object is present in the image by inputting the image to a first neural network model trained to identify an area corresponding to a predetermined object;
obtain posture information of the person object by inputting the image to a second neural network model trained to estimate a posture of an object; and
obtain information regarding the 3D shape model by inputting the information regarding an area in which the person object is present in the image and the posture information of the person object to a third neural network model trained to generate a 3D shape model.

4. The electronic apparatus of claim 1 , wherein the processor is further configured to execute the at least one instruction to:
obtain information regarding the texture to be mapped to the 3D shape model by inputting the image, information regarding the 3D shape model and the identification information to a fourth neural network model trained to obtain information regarding a texture corresponding to a 3D shape model; and
map the texture to the 3D shape model based on the obtained information regarding the texture.

5. The electronic apparatus of claim 1 , wherein the processor is further configured to execute the at least one instruction to:
obtain joint information corresponding to the person object by inputting information regarding the 3D shape model and posture information of the person object to a fifth neural network model trained to obtain joint information; and
generate the 3D avatar based on the 3D shape model to which the texture of the image is mapped and the joint information.

6. The electronic apparatus of claim 1 , wherein the processor is configured to execute the at least one instruction to:
obtain a photographic image as the face image;
obtain the 3D face model of the person object by inputting the photographic image to a sixth neural network model trained to reconstruct a face; and
synthesize the 3D shape model and the 3D face model.

7. The electronic apparatus of claim 1 , wherein the processor is further configured to execute the at least one instruction to, based on a quality of the 3D avatar being equal to or greater than a predetermined quality, control the communication interface to transmit information regarding the 3D avatar to an external apparatus.

8. A method for obtaining a three-dimensional (3D) avatar, the method comprising:
obtaining an image including a person object;
obtaining a face image capturing a face of the person object;
obtaining a 3D face model of the person object based on the face image
obtaining a 3D shape model corresponding to the person object included in the image by mapping a texture of the image to a first area of the 3D shape model to generate a first 3D shape model and mapping the texture of the image to a second area of the 3D shape model based on identification information, the second area being different than the first area to generate a second 3D shape model; and
generating a 3D avatar corresponding to the person object based on the 3D shape model to which the texture of the image is mapped.

9. The method of claim 8 , further comprising:
obtaining identification information for each area of the 3D shape model by performing image segmentation with respect to the first 3D shape model.

10. The method of claim 8 , wherein the obtaining the 3D shape model comprises:
obtaining information regarding an area in which the person object is present in the image by inputting the image to a first neural network model trained to identify an area corresponding to a predetermined object;
obtaining posture information of the person object by inputting the image to a second neural network model trained to estimate a posture of an object; and
obtaining information regarding the 3D shape model by inputting the information regarding an area in which the person object is present in the image and the posture information of the person object to a third neural network model trained to generate a 3D shape model.

11. The method of claim 8 , wherein the mapping the texture of the image to the 3D shape model comprises:
obtaining information regarding the texture to be mapped to the 3D shape model by inputting the image, information regarding the 3D shape model and the identification information to a fourth neural network model trained to obtain information regarding a texture corresponding to a 3D shape model; and
mapping the texture to the 3D shape model based on the obtained information regarding the texture.

12. The method of claim 8 , further comprising:
obtaining joint information corresponding to the person object by inputting information regarding the 3D shape model and posture information of the person object to a fifth neural network model trained to obtain joint information,
wherein the generating a 3D avatar comprises generating the 3D avatar based on the 3D shape model to which the texture of the image is mapped and the joint information.

13. The method of claim 8 , further comprising:
obtaining a photographic image as the face image; and
obtaining the 3D face model of the person object by inputting the photographic image to a sixth neural network model trained to reconstruct a face,
wherein the obtaining the 3D shape model comprises synthesizing the 3D shape model and the 3D face model.

14. A non-transitory computer readable medium for storing computer readable program code or instructions for carrying out operations, when executed by a processor, for obtaining a three-dimensional (3D) avatar, the operations comprising:
obtaining an image including a person object;
obtaining a face image capturing a face of the person object;
obtaining a 3D face model of the person object based on the face image;
obtaining a 3D shape model corresponding to the person object included in the image by mapping a texture of the image to a first area of the 3D shape model to generate a first 3D shape model and mapping the texture of the image to a second area of the 3D shape model based on identification information, the second area being different than the first area to generate a second 3D shape model; and
generating a 3D avatar corresponding to the person object based on the 3D shape model to which the texture of t

Assignees

• Samsung Electronics Co Ltd

Inventors

• Yoo Jisung
• Park Youngjin
• Kim Daehyun
• Moon Boseok
• Lee Yongkoo

Classifications

• G06T2207/30196

  Human being; Person · CPC title

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title

• G06T17/00

  Three-dimensional [3D] modelling for computer graphics · CPC title

• G06V10/22

  by selection of a specific region containing or referencing a pattern; Locating or processing of specific regions to guide the detection or recognition · CPC title

• G06V10/82

  using neural networks · CPC title