Method of generating 3D facial model for an avatar and related device

Patent metadata

Field	Value
Publication number	US-10650564-B1
Application number	US-201916389984-A
Country	US
Kind code	B1
Filing date	Apr 21, 2019
Priority date	Apr 21, 2019
Publication date	May 12, 2020
Grant date	May 12, 2020

Abstract

A method of generating 3D facial geometry for a computing device is disclosed. The method comprises obtaining a 2D image, performing a deep neural network, DNN, operation on the 2D image, to classify each of facial features of the 2D image as texture components and obtain probabilities that the facial feature belong to the texture components, wherein the texture components are represented by 3D face mesh and are predefined in the computing device, and generating a 3D facial model based on a 3D face template predefined in the computing device and the texture component with the highest probability.

First claim

What is claimed is:

1. A method of generating 3D facial geometry for an avatar, for a computing device, the method comprising:
obtaining a 2D image;
performing a landmark detection operation on the 2D image, to obtain at least a facial feature with landmarks;
determining weightings for a plurality of classifications of the facial feature based on relative distances of the landmarks, wherein the plurality of classifications are predefined in the computing device;
performing a deep neural network, DNN, operation on the 2D image, to classify each of facial features of the 2D image as texture components and obtain probabilities that the facial feature belong to the texture components, wherein the texture components are represented by 3D face mesh and are predefined in the computing device; and
generating a 3D facial model based on a 3D face template predefined in the computing device with 3D parameters corresponding to the plurality of classifications of the facial feature, the corresponding weightings, and the texture component with the highest probability.

2. The method of claim 1 , wherein the texture components comprise a lip color, eye bags, an eyebrow texture, a facial hair texture, a hair color, a hair style and a morph target.

3. The method of claim 2 , wherein the eyebrow texture comprises high arch, bushy, thin, straight and soft arch, the facial hair texture comprises goatee beard, sideburn beard, stubble, chin curtain, spade, mutton chops, Old Dutch and hipster, hair color comprises black, brown, blond and gray, the morph target comprises pointy nose, bulbous nose, turned-up tip nose and aquiline nose, and the hair type comprises short, medium, long and bald.

4. The method of claim 1 , wherein the facial features comprise a face shape, eyes, eyebrows, a nose and a mouth.

5. The method of claim 4 , further comprising:
determining at least a classification of the plurality of classifications of the facial feature according to the relative distance of the landmarks.

6. The method of claim 5 , wherein determining the at least a classification of the plurality of classifications of the facial feature according to the relative distance of the landmarks comprises:
determining a width and a length of the facial feature according to the relative distance of the landmarks, to obtain a ratio of the face feature; and
determining the classification of the face feature according to the ratio.

7. The method of claim 1 , wherein determining weightings for the plurality of classifications of the facial feature based on relative distances of the landmarks:
determining a width and a length of the facial feature according to the relative distance of the landmarks, to obtain a ratio of the face feature; and
determining weightings for the plurality of classifications of the facial feature according to the ratio.

8. An avatar simulation system comprising:
a camera, for obtaining a 2D image;
a computing device or a cloud, for generating 3D facial model;
wherein the computing device or the cloud includes:
a processing unit for executing a program; and
a storage unit coupled to the processing unit for storing the program; wherein the program instructs the processing unit to perform the following steps:
obtaining a 2D image;
performing a landmark detection operation on the 2D image, to obtain at least a facial feature with landmarks;
determining weightings for a plurality of classifications of the facial feature based on relative distances of the landmarks, wherein the plurality of classifications are predefined in the computing device;
performing a deep neural network, DNN, operation on the 2D image, to classify each of facial features of the 2D image as texture components and obtain probabilities that the facial feature belong to the texture components, wherein the texture components are represented by 3D face mesh and are predefined in the computing device; and
generating a 3D facial model based on a 3D face template predefined in the computing device with 3D parameters corresponding to the plurality of classifications of the facial feature, the corresponding weightings, and the texture component with the highest probability.

9. The avatar simulation system of claim 8 , wherein the texture components comprise a lip color, eye bags, an eyebrow texture, a facial hair texture, a hair color, a hair style and a morph target.

10. The avatar simulation system of claim 9 , wherein the eyebrow texture comprises high arch, bushy, thin, straight and soft arch, the facial hair texture comprises goatee beard, sideburn beard, stubble, chin curtain, spade, mutton chops, Old Dutch and hipster, hair color comprises black, brown, blond and gray, the morph target comprises pointy nose, bulbous nose, turned-up tip nose and aquiline nose, and the hair type comprises short, medium, long and bald.

11. The avatar simulation system of claim 8 , wherein the facial features comprise a face shape, eyes, eyebrows, a nose and a mouth.

12. The avatar simulation system of claim 11 , wherein the program further instructs the processing unit to perform the following steps:
determining at least a classification of the plurality of classifications of the facial feature according to the relative distance of the landmarks.

13. The avatar simulation system of claim 12 , wherein the program further instructs the processing unit to perform the following steps:
determining a width and a length of the facial feature according to the relative distance of the landmarks, to obtain a ratio of the face feature; and
determining the classification of the face feature according to the ratio.

14. The avatar simulation system of claim 8 , wherein the program further instructs the processing unit to perform the following steps:
determining a width and a length of the facial feature according to the relative distance of the landmarks, to obtain a ratio of the face feature; and
determining weightings for the plurality of classifications of the facial feature according to the ratio.

15. A computing device for generating 3D facial geometry for an avatar comprising:
a processing unit for executing a program; and
a storage unit coupled to the processing unit for storing the program; wherein the program instructs the processing unit to perform the following steps:
obtaining a 2D image;
performing a landmark detection operation on the 2D image, to obtain at least a facial feature with landmarks;
determining weightings for a plurality of classifications of the facial feature based on relative distances of the landmarks, wherein the plurality of classifications are predefined in the computing device;
performing a deep neural network, DNN, operation on the 2D image, to classify each of facial features of the 2D image as texture components and obtain probabilities that the facial feature belong to the texture components, wherein the texture components are represented by 3D face mesh and are predefined in the computing device; and
generating a 3D facial model based on a 3D face template predefined in the computing device with 3D parameters corresponding to the plurality of classifications of the facial feature, the corresponding weightings, and the texture component with the highest probability.

16. The computing device of claim 15 , wherein the texture components comprise a lip color, eye bags, an eyebrow texture, a facial hair texture, a hair color, a hair style and a morph target.

17. The computing device of claim 16 , wherein the eyebrow texture comprises high arch, bushy, thin, straight and soft arch, the faci

Assignees

• Xrspace Co Ltd

Inventors

• Lin Ting-Chieh
• Chou Shih-Chieh

Classifications

• G06T17/20

  Finite element generation, e.g. wire-frame surface description, {tesselation} · CPC title

• G06K9/6267

  Physics · mapped topic

• G06K9/00288

  Physics · mapped topic

• G06N3/04

  Architecture, e.g. interconnection topology · CPC title

• G06T13/40 Primary

  of characters, e.g. humans, animals or virtual beings · CPC title