Face hallucination using convolutional neural networks

Patent metadata

Field	Value
Publication number	US-9405960-B2
Application number	US-201414375683-A
Country	US
Kind code	B2
Filing date	Jun 17, 2014
Priority date	Jun 17, 2014
Publication date	Aug 2, 2016
Grant date	Aug 2, 2016

Abstract

Face hallucination using a bi-channel deep convolutional neural network (BCNN), which can adaptively fuse two channels of information. In one example, the BCNN is implemented to extract high level features from an input image. The extracted high level features are combined with low level details in the input image to produce the higher resolution image. Preferably, a proper coefficient is obtained to adaptively combine the high level features and the low level details.

First claim

What is claimed is:

1. A system for generating higher resolution output face images from input face images, the system comprising:
a convolutional neural network (CNN) that generates a face representation of an input face image, the CNN including convolution, non-linearity and down-sampling;
a face hallucinator that generates a hallucinated face image from the face representation, the hallucinated face image having a higher resolution than the input face image;
a coefficient estimator that generates a coefficient from the face representation; and
a face combiner that combines the hallucinated face image with an up-sampled version of the input face image to produce an output face image, wherein the face combiner generates the output face image as a linear combination of the hallucinated face image and the up-sampled version of the input face image, and the coefficient determines the linear combination,
wherein the output face image has more texture than the hallucinated face image.

2. The system of claim 1 wherein the face representation is a representation of features in the input face image, and the CNN extracts the features from the input face image.

3. The system of claim 2 wherein the CNN comprises two or more CNN layers in a cascade progressing from lower level features to higher level features.

4. The system of claim 1 wherein the face hallucinator is a neural network that generates the hallucinated face image from the face representation.

5. The system of claim 4 wherein the face hallucinator is a fully-connected neural network.

6. The system of claim 1 wherein the coefficient estimator is a neural network that generates the coefficient from the face representation.

7. The system of claim 6 wherein the coefficient estimator is a fully-connected neural network.

8. The system of claim 1 wherein the up-sampling is based on bicubic interpolation.

9. The system of claim 1 wherein the output face image contains at least four times a number of pixels in the input face image.

10. The system of claim 1 wherein the output face image is at least 100×100 pixels.

11. The system of claim 1 wherein the input face image is not more than 50×50 pixels.

12. The system of claim 1 wherein the output face image compensates for Gaussian blur in the input face image.

13. The system of claim 1 wherein the output face image compensates for motion blur in the input face image.

14. The system of claim 1 further comprising:
a landmark detection module that detects face landmarks from the output face image.

15. The system of claim 1 wherein:
the coefficient estimator is a fully-connected neural network; and
the face hallucinator is a fully-connected neural network that generates the hallucinated face image from the face representation.

16. The system of claim 15 wherein the output face image contains at least four times a number of pixels in the input face image.

17. A method for generating higher resolution output face images from input face images, the method comprising:
generating a face representation of an input face image;
generating a hallucinated face image from the face representation, the hallucinated face image having a higher resolution than the input face image;
generating a coefficient from the face representation;
up-sampling the input face image; and
combining the hallucinated face image with the up-sampled input face image to produce an output face image, wherein the output face image is generated as a linear combination of the hallucinated face image and the up-sampled input face image, and the coefficient determines the linear combination,
wherein the output face image has more texture than the hallucinated face image, and
wherein the method is performed with one or more processors.

18. A non-transitory computer readable medium configured to store program code, the program code comprising instructions for generating higher resolution output face images from input face images, the instructions when executed by a processor cause the processor to:
generate a face representation of an input face image;
generate a hallucinated face image from the face representation, the hallucinated face image having a higher resolution than the input face image;
generate a coefficient from the face representation;
up-sample the input face image; and
combine the hallucinated face image with the up-sampled input face image to produce an output face image, wherein the output face image is generated as a linear combination of the hallucinated face image and the up-sampled input face image, and the coefficient determines the linear combination,
wherein the output face image has more texture than the hallucinated face image.

Assignees

• Beijing Kuangshi Tech Co Ltd

Inventors

• Yin Qi
• Cao Zhimin
• Zhou Erjin

Classifications

• G06V40/164 Primary

  using holistic features · CPC title

• G06V40/16

  Human faces, e.g. facial parts, sketches or expressions · CPC title

• G06K9/00221 Primary

  Physics · mapped topic

• G06K9/66

  Physics · mapped topic

• G06V40/171

  Local features and components; Facial parts (eye characteristics G06V40/18); Occluding parts, e.g. glasses; Geometrical relationships · CPC title