Method and apparatus for generating three-dimensional model, computer device and storage medium

Patent metadata

Field	Value
Publication number	US-11475624-B2
Application number	US-202117552621-A
Country	US
Kind code	B2
Filing date	Dec 16, 2021
Priority date	Jun 30, 2020
Publication date	Oct 18, 2022
Grant date	Oct 18, 2022

Abstract

Provided are a method and apparatus for generating a three-dimensional model. The method includes following. A first image containing a first face is acquired. First point cloud data including contour information of the first face is determined based on the first image. First albedo information of the first face and second point cloud data including detail information of the first face are determined based on the first point cloud data and the first image. A three-dimensional model of the first face is generated based on the first albedo information and the second point cloud data.

First claim

The invention claimed is:

1. A method for generating a three-dimensional model, comprising:
acquiring a first image containing a first face;
determining, based on the first image, first point cloud data comprising contour information of the first face;
determining, based on the first point cloud data and the first image, first albedo information of the first face by using a first neural network trained by weak supervision and determining, based on the first point cloud data and the first image, second point cloud data comprising detail information of the first face by using a second neural network trained by weak supervision; and
generating a three-dimensional model of the first face based on the first albedo information and the second point cloud data;
wherein the first neural network is trained by:
acquiring a second image containing a second face, and obtaining third point cloud data comprising contour information of the second face based on the second image;
obtaining a second facial texture map of the second face based on the third point cloud data;
performing albedo prediction and illumination prediction for the second facial texture map by using a third neural network to be trained, to obtain second albedo information of the second face and first illumination information of the second face;
performing albedo prediction for the second facial texture map by using the first neural network to be trained, to obtain third albedo information of the second face; and
using the second albedo information, the first illumination information, and the third albedo information to jointly train the first neural network to be trained and the third neural network to be trained, to obtain a trained first neural network.

2. The method for generating the three-dimensional model according to claim 1 , wherein determining, based on the first point cloud data and the first image, the first albedo information of the first face by using the first neural network trained by weak supervision and determining, based on the first point cloud data and the first image, the second point cloud data comprising the detail information of the first face by using the second neural network trained by weak supervision comprises:
determining a first facial texture map of the first face based on the first point cloud data and the first image;
performing albedo information prediction for the first facial texture map by using the first neural network to obtain the first albedo information of the first face; and
performing facial detail prediction for the first facial texture map by using the second neural network, and obtaining the second point cloud data of the first face based on a result of the facial detail prediction.

3. The method for generating the three-dimensional model according to claim 2 , wherein the first point cloud data comprises:
three-dimensional coordinate values of a plurality of first point cloud points constituting the first face in a camera coordinate system, and
connection relationship information between different ones of the plurality of first point cloud points.

4. The method for generating the three-dimensional model according to claim 3 , wherein determining the first facial texture map of the first face based on the first point cloud data and the first image comprises:
aligning the plurality of first point cloud points with a plurality of first pixels in the first image respectively based on the three-dimensional coordinate values of the plurality of first point cloud points in the camera coordinate system;
determining second pixel values corresponding to each of a plurality of face patches constituting the first face based on a result of the alignment, the connection relationship information between the different ones of the plurality of first point cloud points, and first pixel values of the plurality of first pixels in the first image, wherein each of the plurality of face patches is constituted by at least three first point cloud points having a connection relationship with each other; and
generating the first facial texture map based on the second pixel values corresponding to each of the plurality of face patches.

5. The method for generating the three-dimensional model according to claim 2 , wherein the result of the facial detail prediction comprises first facial detail information of the first face; and
obtaining the second point cloud data of the first face based on the result of the facial detail prediction comprises:
fusing the first point cloud data and the first facial detail information to obtain the second point cloud data.

6. The method for generating the three-dimensional model according to claim 5 , wherein the first facial detail information comprises: a movement distance of each of a plurality of first point cloud points in a first normal vector direction corresponding to the first point cloud point; and
fusing the first point cloud data and the first facial detail information to obtain the second point cloud data comprises:
for each of the plurality of first point cloud points, adjusting a position of the first point cloud point in a camera coordinate system based on the movement distance of the first point cloud point in the first normal vector direction corresponding to the first point cloud point, and based on three-dimensional coordinate values of the first point cloud point in the camera coordinate system; and
obtaining the second point cloud data based on results of adjusting the plurality of first point cloud points.

7. The method for generating the three-dimensional model according to claim 1 , wherein using the second albedo information, the first illumination information, and the third albedo information to jointly train the first neural network to be trained and the third neural network to be trained, to obtain the trained first neural network comprises:
determining a first loss of the third neural network to be trained, by using the second albedo information and the first illumination information;
determining a second loss of the first neural network to be trained, by using the second albedo information, the first illumination information, and the third albedo information;
updating, based on the first loss, network parameters of the third neural network to be trained, and updating, based on the second loss, network parameters of the first neural network to be trained; and
obtaining the trained first neural network by a plurality of times of updating the network parameters of the third neural network to be trained and the network parameters of the first neural network to be trained.

8. The method for generating the three-dimensional model according to claim 7 , wherein determining the first loss of the third neural network to be trained, by using the second albedo information and the first illumination information comprises:
determining, based on the third point cloud data, second normal vectors respectively corresponding to a plurality of third point cloud points in the third point cloud data;
generating a third facial texture map of the second face based on the second normal vectors, the second albedo information, and the first illumination information; and
generating the first loss based on the second facial texture map and the third facial texture map.

9. The method for generating the three-dimensional model according to claim 8 , wherein determining the second loss of the first neural network to be trained, by using the second albedo information, the first illumination information, and the third albedo information comprises:
determining a first sub-loss based on the second albedo information and the third albedo information;
generating a fourth facial texture map of the s

Assignees

• Beijing Sensetime Tech Development Co Ltd

Inventors

• Wang Pengrui
• Lin Chunze
• Wang Quan
• Qian Chen

Classifications

• G06F18/25

  Fusion techniques · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G06T2207/20081

  Training; Learning · CPC title

• G06N3/08

  Learning methods · CPC title

• G06V10/82

  using neural networks · CPC title