Facial image generating method, facial image generating apparatus, and facial image generating device

Patent metadata

Field	Value
Publication number	US-10832034-B2
Application number	US-201715814101-A
Country	US
Kind code	B2
Filing date	Nov 15, 2017
Priority date	Nov 16, 2016
Publication date	Nov 10, 2020
Grant date	Nov 10, 2020

Abstract

A facial image generating method, a facial image generating apparatus, and a facial image generating device are proposed. The method comprises: linking an M-dimensional facial feature vector with an N-dimensional demanded feature vector to generate a synthesized feature vector; and generating a synthesized facial image by use of a deep convolutional network for facial image generation and based on the synthesized feature vector. The method further comprises: generating a demand satisfaction score based on the synthesized facial image and the demanded feature vector by use of a deep convolutional network for demand determination; and updating parameters of the deep convolutional network for facial image generation and the deep convolutional network for demand determination based on the demand satisfaction score. A facial image is generated based on a facial feature vector and a demanded feature vector, a facial image with a specific feature can be generated without using the three-dimensional model.

First claim

What is claimed is:

1. A facial image generating method, comprising:
generating an M-dimensional facial feature vector, M being an integer larger than one;
linking the M-dimensional facial feature vector with an N-dimensional demanded feature vector to generate an (M+N)-dimensional synthesized feature vector, N being an integer larger than or equal to one; and
generating a synthesized facial image by use of a deep convolutional network for facial image generation and based on the synthesized feature vector,
wherein generating the facial feature vector comprises:
extracting the facial feature vector from a given facial image by use of a deep convolutional network for facial feature extraction; or
randomly generating the facial feature vector,

wherein the deep convolutional network for facial feature extraction comprises:
P layers of convolutional neural network and at least one layer of fully connected neural network, P being an integer larger than or equal to two,
wherein a first layer of convolutional neural network is used to receive the given facial image, the at least one layer of fully connected neural network is used to receive images outputted from a P-th layer of convolutional neural network and generate the facial feature vector.

2. The facial image generating method according to claim 1 , wherein the deep convolutional network for facial image generation comprises at least one layer of fully connected neural network and K layers of integrated convolutional neural network, each layer of integrated convolutional neural network comprising an amplification network and J layers of convolutional neural network, K being an integer larger than or equal to two, and J being an integer larger than or equal to two.

3. The facial image generating method according to claim 2 , wherein generating the synthesized facial image by use of the deep convolutional network for facial image generation and based on the synthesized feature vector comprises:
generating initial synthesized images by use of the at least one layer of fully connected neural network and based on the synthesized feature vector;
receiving the initial synthesized images and generating synthesized images of a first layer by use of the first layer of integrated convolutional neural network, a number of the synthesized images of the first layer being smaller than a number of the initial synthesized images; and
receiving synthesized images outputted from a (k−1)-th layer of integrated convolutional neural network and generating synthesized images of a k-th layer by use of the k-th layer of integrated convolutional neural network, k being an integer larger than or equal to two and smaller than or equal to K, a size of the synthesized images of the k-th layer being larger than a size of the synthesized images of the (k−1)-th layer, and a number of the synthesized images of the k-th layer being smaller than a number of the synthesized images of the (k−1)-th layer,
wherein synthesized images of a K-th layer as outputted from the K-th layer of integrated convolutional neural network is taken as the synthesized facial image.

4. The facial image generating method according to claim 2 , wherein generating the synthesized facial image by use of the deep convolutional network for facial image generation and based on the synthesized feature vector comprises:
generating initial synthesized images by use of the at least one layer of fully connected neural network and based on the synthesized feature vector;
receiving the initial synthesized images and N initial mapped images generated as mapping from the N-dimensional demanded feature vector and generating synthesized images of a first layer by use of the first layer of integrated convolutional neural network, a size of the initial synthesized images being the same as a size of the initial mapped images, a number of the synthesized images of the first layer being smaller than a number of the initial synthesized images; and
receiving synthesized images outputted from a (k−1)-th layer of integrated convolutional neural network and N mapped images of the (k−1)-th layer generated as mapping from the N-dimensional demanded feature vector and generating synthesized images of a k-th layer by use of the k-th layer of integrated convolutional neural network, a size of the synthesized images of the (k−1)-th layer being the same as a size of the mapped images of the (k−1)-th layer, a size of the synthesized images of the k-th layer being larger than a size of the synthesized images of the (k−1)-th layer, and a number of the synthesized images of the k-th layer being smaller than a number of the synthesized images of the (k−1)-th layer,
wherein each dimension of the N-dimensional demanded feature vector is mapped as one of the N initial mapped images and also mapped as one of N mapped images of the (k−1)-th layer, k being an integer larger than or equal to two and smaller than or equal to K,
wherein synthesized images of a K-th layer as outputted from the K-th layer of integrated convolutional neural network is taken as the synthesized facial image.

5. The facial image generating method according to claim 1 , further comprising:
generating a demand satisfaction score based on the synthesized facial image and the demanded feature vector and by use of a deep convolutional network for demand determination; and
updating parameters of the deep convolutional network for facial image generation and the deep convolutional network for demand determination based on the demand satisfaction score.

6. The facial image generating method according to claim 5 , wherein in the case of extracting the facial feature vector from a given facial image, the facial image generating method further comprises:
generating a face matching score based on the synthesized facial image and the given facial image and by use of a first deep convolutional network for face determination; and
updating parameters of the deep convolutional network for facial feature extraction, the deep convolutional network for facial image generation, and the first deep convolutional network for face determination based on the face matching score.

7. The facial image generating method according to claim 5 , wherein in the case of randomly generating the facial feature vector, the facial image generating method further comprises:
generating a face satisfaction score based on the synthesized facial image and by use of a second deep convolutional network for face determination; and
updating parameters of the deep convolutional network for facial image generation and the second deep convolutional network for face determination based on the face satisfaction score.

8. The facial image generating method according to claim 6 , wherein the first deep convolutional network for face determination comprises a first to-be-determined feature vector extraction network, a second to-be-determined feature vector extraction network, and a fully connected neural network,
wherein the first to-be-determined feature vector extraction network is used to extract a first to-be-determined feature vector from the given facial image;
the second to-be-determined feature vector extraction network is used to extract a second to-be-determined feature vector from the synthesized facial image;
the fully connected neural network is used to generate the face matching score based on the first to-be-determined feature vector and the second to-be-determined feature vector,
wherein parameters of the first to-be-determined feature vector extraction network are the same as parameters of the second to-be-determined feature vector extraction network, a dimension number of the first to-be-determined feature vector is the same as a dimension numbe

Assignees

• Beijing Kuangshi Tech Co Ltd
• Megvii Beijing Technology Co Ltd

Inventors

• Liu Yu
• Zhou Erjin

Classifications

• G06N3/045

  Combinations of networks · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06N3/0475

  Generative networks · CPC title

• G06V10/82

  using neural networks · CPC title