Face super-resolution realization method and apparatus, electronic device and storage medium

Patent metadata

Field	Value
Publication number	US-11710215-B2
Application number	US-202117208651-A
Country	US
Kind code	B2
Filing date	Mar 22, 2021
Priority date	Jun 17, 2020
Publication date	Jul 25, 2023
Grant date	Jul 25, 2023

Abstract

The present application discloses a face super-resolution realization method and apparatus, an electronic device and a storage medium, and relate to fields of face image processing and deep learning. The specific implementation solution is as follows: a face part in a first image is extracted; the face part is input into a pre-trained face super-resolution model to obtain a super-sharp face image; a semantic segmentation image corresponding to the super-sharp face image is acquired; and the face part in the first image is replaced with the super-sharp face image, by utilizing the semantic segmentation image, to obtain a face super-resolution image.

First claim

What is claimed is:

1. A face super-resolution realization method, comprising:
extracting a face part in a first image;
inputting the face part into a pre-trained face super-resolution model to obtain a super-sharp face image;
acquiring a semantic segmentation image corresponding to the super-sharp face image; and
replacing the face part in the first image with the super-sharp face image, by utilizing the semantic segmentation image, to obtain a face super-resolution image;
wherein the method further comprises:
detecting face key point information in an original image; and
enlarging the original image to obtain the first image, and obtaining face key point information in the first image, wherein the face key point information in the first image comprises information obtained by performing image enlargement processing on the face key point information in the original image;
wherein extracting the face part in the first image comprises: calculating a conversion matrix for realizing face alignment by utilizing the face key point information in the first image; and extracting the face part in the first image by utilizing the conversion matrix;
wherein the replacing the face part in the first image with the super-sharp face image by utilizing the semantic segmentation image, comprises: placing the super-sharp face image at a first position by utilizing an inverse matrix of the conversion matrix, wherein the first position is a position of the face part in the first image; and taking the semantic segmentation image as a mask image, and fusing the super-sharp face image placed at the first position with the first image by utilizing the mask image, to replace the face part in the first image.

2. The method of claim 1 , wherein detecting the face key point information in the original image comprises:
inputting the original image into a pre-trained face key point detection model to obtain the face key point information in the original image.

3. The method of claim 1 , wherein the face super-resolution model is a generative adversarial network (GAN) model.

4. The method of claim 1 , wherein acquiring the semantic segmentation image corresponding to the super-sharp face image comprises:
inputting the super-sharp face image into a pre-trained face segmentation model to obtain a segmented image; and
determining the semantic segmentation image corresponding to the super-sharp face image by utilizing the segmented image.

5. The method of claim 2 , wherein extracting the face part in the first image comprises:
calculating a conversion matrix for realizing face alignment by utilizing the face key point information in the first image; and
extracting the face part in the first image by utilizing the conversion matrix.

6. A face super-resolution realization apparatus, comprising:
a processor and a memory for storing one or more computer programs executable by the processor,
wherein when executing at least one of the computer programs, the processor is configured to:
extract a face part in a first image;
input the face part into a pre-trained face super-resolution model to obtain a super-sharp face image;
acquire a semantic segmentation image corresponding to the super-sharp face image; and
replace the face part in the first image with the super-sharp face image, by utilizing the semantic segmentation image, to obtain a face super-resolution image;
wherein when executing at least one of the computer programs, the processor is further configured to: detect face key point information in the original image; and enlarge the original image to obtain the first image, and obtain face key point information in the first image, wherein the face key point information in the first image comprises information obtained by performing image enlargement processing on the face key point information in the original image;
wherein when executing at least one of the computer programs, the processor is further configured to: calculate a conversion matrix for realizing face alignment by utilizing the face key point information in the first image; and extract the face part in the first image by utilizing the conversion matrix;
wherein when executing at least one of the computer programs, the processor is further configured to: place the super-sharp face image at a first position by utilizing an inverse matrix of the conversion matrix, wherein the first position is a position of the face part in the first image; and take the semantic segmentation image as a mask image, and fuse the super-sharp face image placed at the first position with the first image by utilizing the mask image, to replace the face part in the first image.

7. The apparatus of claim 6 , wherein when executing at least one of the computer programs, the processor is further configured to: input the original image into a pre-trained face key point detection model to obtain the face key point information in the original image.

8. The apparatus of claim 6 , wherein the face super-resolution model is a generative adversarial network (GAN) model.

9. The apparatus of claim 6 , wherein when executing at least one of the computer programs, the processor is further configured to:
input the super-sharp face image into a pre-trained face segmentation model to obtain a segmented image; and
determine the semantic segmentation image corresponding to the super-sharp face image by utilizing the segmented image.

10. A non-transitory computer-readable storage medium storing computer instructions thereon, the computer instructions causing a computer to execute the method of claim 1 .

11. A non-transitory computer-readable storage medium storing computer instructions thereon, the computer instructions causing a computer to execute the method of claim 2 .

Assignees

• Beijing Baidu Netcom Sci & Tech Co Ltd

Inventors

• Hu Tianshu
• Liu Jiaming
• Hong Zhibin

Classifications

• G06N3/09

  Supervised learning · CPC title

• G06N3/094

  Adversarial learning · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/0475

  Generative networks · CPC title

• G06T3/4046 Primary

  using neural networks · CPC title