Face modeling method and apparatus, electronic device and computer-readable medium

Patent metadata

Field	Value
Publication number	US-11625896-B2
Application number	US-201917267723-A
Country	US
Kind code	B2
Filing date	Aug 9, 2019
Priority date	Sep 26, 2018
Publication date	Apr 11, 2023
Grant date	Apr 11, 2023

Abstract

A face modeling method and apparatus, an electronic device and a computer-readable medium. Said method comprises: acquiring multiple depth images, the multiple depth images being obtained by photographing a target face at different irradiation angles; performing alignment processing on the multiple depth images to obtain a target point cloud image; using the target point cloud image to construct a three-dimensional model of the target face. The present disclosure alleviates the technical problems of poor robustness and low precision of the three-dimensional model constructed according to the three-dimensional model constructing method.

First claim

What is claimed is:

1. A face modeling method, comprising:
acquiring a plurality of frames of depth images, wherein the plurality of frames of depth images are obtained by shooting a target face with a depth camera at different shooting angles;
performing depth image alignment processing on the plurality of frames of depth images to obtain a target point cloud image, wherein the target point cloud image comprises a plurality of three-dimensional vertices, and both a three-dimensional coordinate and a normal vector of each three-dimensional vertex of the plurality of three-dimensional vertices; and
constructing a three-dimensional model of the target face by using the target point cloud image,
wherein performing depth image alignment processing on the plurality of frames of depth images to obtain the target point cloud image, comprises:
transforming each frame of a depth image into a three-dimensional point cloud image, wherein the three-dimensional point cloud image comprises three-dimensional coordinates and normal vectors of respective three-dimensional vertices in a point cloud; and
performing vertex alignment processing on the respective three-dimensional vertices in respective three-dimensional point cloud images to obtain the target point cloud image, and
wherein performing vertex alignment processing on the respective three-dimensional vertices in respective three-dimensional point cloud images to obtain the target point cloud image, comprises:
aiming at an M-th frame of the depth image of the plurality of frames of depth images, performing point cloud matching on the three-dimensional point cloud image of the M-th frame of depth image and a predicted point cloud image, and determining pose information of the M-th frame of the depth image shot by the depth camera according to a point cloud matching result;
determining position information of respective vertices in the three-dimensional point cloud image of the M-th frame of the depth image in real space based on the pose information;
fusing the three-dimensional point cloud image of the M-th frame of depth image into the predicted point cloud image by using the position information; and
repeatedly executing the above steps until a three-dimensional point cloud image of an N-th frame of depth image is fused into a predicted point cloud image, after aligning three-dimensional point cloud images of previous (N-1) frames of depth images, to obtain the target point cloud image,
wherein N is an amount of depth images, M is less than or equal to N, the predicted point cloud image is a point cloud image after aligning three-dimensional point cloud images of previous (M-1) frames of depth images, and the M-th frame of depth image is an M-th frame of depth image taken by the depth camera on the target face.

2. The method according to claim 1 , wherein the three-dimensional model comprises a plurality of three-dimensional vertices, and one three-dimensional vertex corresponds to one vertex in the target point cloud image; and
constructing the three-dimensional model of the target face by using the target point cloud image, comprises:
determining vertices, which corresponds to respective three-dimensional vertices in the three-dimensional model, in the plurality of three-dimensional vertices of the target point cloud image; and
constructing the three-dimensional model of the target face in an original facial model based on three-dimensional coordinates and normal vectors of the vertices which corresponds to respective three-dimensional vertices in the three-dimensional model.

3. The method according to claim 1 , further comprising:
acquiring a plurality of frames of RGB images, wherein the plurality of frames of RGB images are obtained by shooting the target face at different shooting angles;
performing texture stitching on the plurality of frames of RGB images to obtain a target texture map; and
rendering the three-dimensional model by using the target texture map to obtain the three-dimensional model that is rendered.

4. The method according to claim 3 , wherein performing texture stitching on the plurality of frames of RGB images to obtain the target texture map, comprises:
determining texture features of respective vertices of the three-dimensional model in the plurality of frames of RGB images;
performing UV unfolding processing on the three-dimensional model to obtain an unfolded image, wherein the unfolded image comprises a plurality of two-dimensional coordinates, and one two-dimensional coordinate corresponds to one vertex in the three-dimensional model; and
determining texture features corresponding to respective two-dimensional coordinates in the unfolded image, so as to obtain the target texture map.

5. The method according to claim 4 , wherein determining texture features of respective vertices of the three-dimensional model in the plurality of frames of RGB images, comprises:
determining texture features corresponding to respective vertices of the three-dimensional model in the plurality of frames of RGB images, wherein one vertex corresponds to one or more texture features;
classifying the respective vertices according to an amount of texture features corresponding to the respective vertices to obtain a first classification group and a second classification group, wherein each vertex in the first classification group corresponds to one texture feature, and each vertex in the second classification group corresponds to a plurality of texture features;
taking texture features corresponding to respective vertices in the first classification group as texture features of vertices, which correspond to the respective vertices in the first classification group, in the three-dimensional model; and
determining a target texture feature in the plurality of texture features corresponding to each vertex in the second classification group, and taking the target texture feature as a texture feature of a vertex, which corresponds to each vertex in the second classification group, in the three-dimensional model.

6. The method according to claim 5 , wherein determining the target texture feature in the plurality of texture features corresponding to each vertex in the second classification group, comprises:
calculating a normal offset visual angle of a vertex Ai in the second classification group relative to a target camera, wherein the target camera is a camera used for shooting a target RGB image, and the target RGB image is an RGB image, to which a plurality of texture features corresponding to the vertex Ai belong, in the plurality of frames of RGB images, wherein i takes one to I in sequence, and I is an amount of vertices in the second classification group; and
taking a texture feature corresponding to a minimum normal offset visual angle of the normal offset visual angle as a target texture feature of the vertex Ai.

7. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and being capable of running on the processor, wherein the computer program is capable of being executed by the processor to implement the method according to claim 6 .

8. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and being capable of running on the processor, wherein the computer program is capable of being executed by the processor to implement the method according to claim 5 .

9. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and being capable of running on the processor, wherein the computer program is capable of being executed by the processor to implement the method according to claim 4 .

10. The method according to

Assignees

• Beijing Kuangshi Tech Co Ltd

Inventors

• Qiao Liang
• Chen Keqing
• Huang Haibin

Classifications

• G06T17/00 Primary

  Three-dimensional [3D] modelling for computer graphics · CPC title

• G06T2210/56

  Particle system, point based geometry or rendering · CPC title

• G06T2207/30201

  Face · CPC title

• G06T7/55

  from multiple images · CPC title

• G06T3/4038

  Image mosaicing, e.g. composing plane images from plane sub-images · CPC title