Face recognition using larger pose face frontalization

Patent metadata

Field	Value
Publication number	US-10474880-B2
Application number	US-201815888629-A
Country	US
Kind code	B2
Filing date	Feb 5, 2018
Priority date	Mar 15, 2017
Publication date	Nov 12, 2019
Grant date	Nov 12, 2019

Abstract

A face recognition system is provided. The system includes a device configured to capture an input image of a subject. The system further includes a processor. The processor estimates, using a 3D Morphable Model (3DMM) conditioned Generative Adversarial Network, 3DMM coefficients for the subject of the input image. The subject varies from an ideal front pose. The processor produces, using an image generator, a synthetic frontal face image of the subject of the input image based on the input image and the 3DMM coefficients. An area spanning the frontal face of the subject is made larger in the synthetic image than in the input image. The processor provides, using a discriminator, a decision indicative of whether the subject of the synthetic image is an actual person. The processor provides, using a face recognition engine, an identity of the subject in the input image based on the synthetic and input images.

First claim

What is claimed is:

1. A face recognition system, comprising:
an image capture device configured to capture an input image of a subject;
a processor, configured to
estimate, using a three-dimensional Morphable Model (3DMM) conditioned Generative Adversarial Network (GAN), 3DMM coefficients for the subject of the input image, wherein the subject varies from an ideal front pose;
produce, using an image generator, a synthetic frontal face image of the subject of the input image based on the input image and the 3DMM coefficients, wherein an area spanning the frontal face of the subject is made larger in the synthetic frontal face image than in the input image;
provide, using a discriminator, a decision indicative of whether the subject of the synthetic frontal face image is an actual person; and
provide, using a face recognition engine, an identity of the subject in the input image based on the synthetic frontal face image and the input image.

2. The face recognition system of claim 1 , wherein the face recognition engine regularizes the identity of the subject in the input image relative to the synthetic frontal face image.

3. The face recognition system of claim 1 , wherein the discriminator provides the decision by distinguishing the synthetic frontal face image from at least one ground truth frontal face image.

4. The face recognition system of claim 1 , wherein the discriminator is pre-trained to minimize a classification loss between the input image and the synthetic frontal face image.

5. The face recognition system of claim 1 , wherein the discriminator comprises a linear layer that generates a two-dimensional vector, with each of the two dimensions representing a respective probability of the subject of the synthetic frontal face image being the actual person.

6. The face recognition system of claim 1 , wherein the image generator is configured to mislead the discriminator to classify synthetic frontal face images to be real images using an objective function in order to improve both generator performance and face recognition engine performance.

7. The face recognition system of claim 1 , wherein the face recognition engine is pre-trained to maximize a probability of the subject of the input image being classified as a one of a plurality of ground truth identities.

8. The face recognition system of claim 1 , wherein the generator is guided to generate the synthesized frontal face image so as to include an identity-preserved frontal face relative to the subject of the input image.

9. The face recognition system of claim 1 , wherein the image generator generates the synthetic frontal face image using a symmetry loss to account for symmetry variations between opposing face sides.

10. The face recognition system of claim 1 , wherein the 3DMM is configured to define a three-dimensional face shape and texture in a principal component analysis space.

11. The face recognition system of claim 1 , wherein the input image is used by the 3DMM to compensate for a loss of discriminative identity features of the subject.

12. The face recognition system of claim 1 , wherein the 3DMM conditioned GAN uses a 3DMM fitting process to estimate the 3DMM coefficients for the input image.

13. A computer-implemented method for face recognition, comprising:
capturing, by an image capture device, an input image of a subject;
estimating, by a processor using a three-dimensional Morphable Model (3DMM) conditioned Generative Adversarial Network (GAN), 3DMM coefficients for the subject of the input image, wherein the subject varies from an ideal front pose;
producing, by the processor using an image generator, a synthetic frontal face image of the subject of the input image based on the input image and the 3DMM coefficients, wherein an area spanning the frontal face of the subject is made larger in the synthetic frontal face image than in the input image;
providing, by the processor using a discriminator, a decision indicative of whether the subject of the synthetic frontal face image is an actual person; and
providing, by the processor using a face recognition engine, an identity of the subject in the input image based on the synthetic frontal face image and the input image.

14. The computer-implemented method of claim 13 , wherein the face recognition engine regularizes the identity of the subject in the input image relative to the synthetic frontal face image.

15. The computer-implemented method of claim 13 , wherein the discriminator provides the decision by distinguishing the synthetic frontal face image from at least one ground truth frontal face image.

16. The computer-implemented method of claim 13 , wherein the discriminator is pre-trained to minimize a classification loss between the input image and the synthetic frontal face image.

17. The computer-implemented method of claim 13 , wherein the discriminator comprises a linear layer that generates a two-dimensional vector, with each of the two dimensions representing a respective probability of the subject of the synthetic frontal face image being the actual person.

18. The computer-implemented method of claim 13 , wherein the generator is configured to mislead the discriminator to classify synthetic frontal face images to be real images using an objective function in order to improve both generator performance and face recognition engine performance.

19. The computer-implemented method of claim 13 , wherein the face recognition engine is pre-trained to maximize a probability of the subject of the input image being classified as a one of a plurality of ground truth identities.

20. A computer program product for face recognition, the computer program product comprising a non-transitory computer readable storage medium having program instructions embodied therewith, the program instructions executable by a computer to cause the computer to perform a method comprising:
capturing, by an image capture device, an input image of a subject;
estimating, by a processor using a three-dimensional Morphable Model (3DMM) conditioned Generative Adversarial Network (GAN), 3DMM coefficients for the subject of the input image, wherein the subject varies from an ideal front pose;
producing, by the processor using an image generator, a synthetic frontal face image of the subject of the input image based on the input image and the 3DMM coefficients, wherein an area spanning the frontal face of the subject is made larger in the synthetic frontal face image than in the input image;
providing, by the processor using a discriminator, a decision indicative of whether the subject of the synthetic frontal face image is an actual person; and
providing, by the processor using a face recognition engine, an identity of the subject in the input image based on the synthetic frontal face image and the input image.

Assignees

• Nec Lab America Inc
• Nec Corp

Inventors

• Yu Xiang
• Sohn Kihyuk
• Chandraker Manmohan

Classifications

• G06V10/82

  using neural networks · CPC title

• G06V20/647 Primary

  by matching two-dimensional images to three-dimensional objects · CPC title

• G06V40/172

  Classification, e.g. identification · CPC title

• G06F18/214

  Generating training patterns; Bootstrap methods, e.g. bagging or boosting · CPC title

• G06V10/454

  Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN] · CPC title