Methods and systems for verifying face images based on canonical images

Patent metadata

Field	Value
Publication number	US-10037457-B2
Application number	US-201615282851-A
Country	US
Kind code	B2
Filing date	Sep 30, 2016
Priority date	Apr 11, 2014
Publication date	Jul 31, 2018
Grant date	Jul 31, 2018

Abstract

Disclosed herein are a system and method for verifying face images based on canonical images. The method includes: retrieving, from a plurality of face images of an identity, a face image with a smallest frontal measurement value as a representative image of the identity; determining parameters of an image reconstruction network based on mappings between the retrieved representative image and the plurality of face images of the identity; reconstructing, by the image reconstruction network with the determined parameters, at least two input face images into corresponding canonical images respectively; and comparing the reconstructed canonical images to verify whether they belong to a same identity, where the representative image is a frontal image and the frontal measurement value represents symmetry of each face image and sharpness of the image. Thus, canonical face images can be reconstructed using only 2D information from face images under an arbitrary pose and lighting condition.

First claim

What is claimed is:

1. A method for verifying face images based on canonical images, the method comprising:
retrieving, from a plurality of face images of an identity, a face image with a smallest frontal measurement value as a representative image of the identity;
determining parameters of an image reconstruction network based on mappings between the retrieved representative image and the plurality of face images of the identity;
reconstructing, by the image reconstruction network with the parameters of the image reconstruction network, at least two input face images into corresponding canonical images; and
comparing the reconstructed canonical images to verify whether the reconstructed canonical images belong to a same identity,
wherein the representative image is a frontal image and a frontal measurement value represents symmetry of a face image and sharpness of the face image.

2. The method according to claim 1 , further comprising:
adjusting, after reconstructing the at least two input face images, the parameters of the image reconstruction network based on transformation between an input face image and the corresponding reconstructed canonical image.

3. The method according to claim 1 , wherein the image reconstruction network comprises a plurality of layers of sub-networks, and determining the parameters of the image reconstruction network comprises:
determining preliminary parameters of each layer of the image reconstruction network based on the mappings by inputting an image training set, wherein an output of a previous layer of sub-network is inputted into a current layer of sub-network during the determining;
comparing an output of a last layer of sub-network and an expected target to obtain an error therebetween; and
fine-tuning, based on the obtained error, the preliminary parameters to concrete the parameters of the image reconstruction network.

4. The method according to claim 1 , further comprising:
acquiring, before comparing the reconstructed canonical images, a similarity between any two reconstructed canonical images to determine parameters of an image verification network.

5. The method according to claim 4 , further comprising:
selecting one or more facial components from each of the reconstructed canonical images respectively to form one or more facial component pairs, each facial component pair including facial components corresponding to same face regions in the reconstructed canonical images respectively; and
acquiring a similarity between the one or more facial component pairs to determine the parameters of the image verification network.

6. The method according to claim 1 , wherein the frontal measurement value M(Y i ) is formulated by rule of:
M ( Y i )=∥ Y i P−Y i Q∥ F 2 −λ∥Y i ∥*

where Y i ∈D i denotes a face image in a set of face images D i ,
λ is a constant coefficient,
∥⋅∥ F is the Frobenius norm,
λ⋅∥* is the nuclear norm, and
P and Q represent two constant matrixes with P=diag([1 a , 0 a ]) and Q=diag ([0 a , 1 a ]), where diag(⋅) indicates a diagonal matrix.

7. A system for verifying face images based on canonical images, the system comprising:
a memory storing one or more computer-executable components;
a processor electrically communicated with the memory to execute the computer-executable components to perform operations of the system, wherein the computer-executable components comprise:
a retrieving component configured to retrieve, from a plurality of face images of an identity, a face image with a smallest frontal measurement value as a representative image of the identity;
an image reconstruction component configured to reconstruct input face images into corresponding canonical images;
a determining component configured to determine parameters of the image reconstruction component, wherein the parameters are determined based on mappings between the representative image retrieved by the retrieving component and the plurality of face images of the identity; and
a comparing component configured to compare the canonical images reconstructed by the image reconstruction component to verify whether the canonical images belong to a same identity,
wherein the representative image is a frontal image and a frontal measurement value represents symmetry of a face image and sharpness of the face image.

8. The system according to claim 7 , wherein the determining component is further configured to adjust the parameters of the image reconstruction component based on transformation between an input face image and the corresponding reconstructed canonical image.

9. The system according to claim 8 , wherein the computer-executable components further comprise:
an acquiring component configured to acquire a similarity between any two reconstructed canonical images; and
an image verification component configured to verify whether a pair of face images belong to a same identity,
wherein the determining component is further configured to determine parameters of the image verification component based on the similarity between any two reconstructed canonical images acquired by the acquiring component.

10. The system according to claim 9 , wherein the computer-executable components further comprise:
a selecting component configured to select one or more facial components from each of the reconstructed canonical images respectively to form one or more facial component pairs, each facial component pair including facial components corresponding to same face regions in the reconstructed canonical images respectively;
wherein the acquiring component is further configured to acquire a similarity between the one or more facial component pairs, and
wherein the determining component is further configured to determine the parameters of the image verification component based on the similarity between the one or more facial component pairs acquired by the acquiring component.

11. The system according to claim 9 , wherein the image verification component is formed as a multilayer image verification neural network.

12. The system according to claim 7 , wherein the image reconstruction component is formed as a multilayer image reconstruction neural network.

13. The system according to claim 12 , wherein:
the image reconstruction neural network comprises a plurality of layers of sub-networks, and the determining component is further configured to determine preliminary parameters of each layer of the image reconstruction neural network based on the mappings by inputting an image training set, wherein an output of a previous layer of sub-network is inputted into a current layer of sub-network during the determining; and
the determining component is further configured to compare an output of a last layer of sub-network and an expected target to obtain an error therebetween; and based on the obtained error, fine-tune the preliminary parameters to concrete the parameters of the image reconstruction neural network.

14. The system according to claim 7 , wherein the frontal measurement value is formulated by the following equation:
M ( Y )=∥ Y i P−Y i Q∥ F 2 −λ∥Y i ∥*

where Y i ∈D i denotes a face image in a set of face images D i ,
λ is a constant coefficient,
∥⋅∥ F is the Frobenius norm,
∥⋅∥* is the nuclear norm, and
P and Q represent two constant matrixes with P=diag ([1 a ,0 a ]) and Q=diag([0 a ,1 a ]), where diag(⋅) indicates a diagonal matrix.

15. A method for verifying face images based on canonical images, the method comprising:
retrieving a face image with a smallest frontal measurement value from a plurality

Assignees

• Beijing Sensetime Tech Development Co Ltd

Inventors

• Tang Xiaoou
• Zhu Zhenyao
• Luo Ping
• Wang Xiaogang

Classifications

• G06V40/171 Primary

  Local features and components; Facial parts (eye characteristics G06V40/18); Occluding parts, e.g. glasses; Geometrical relationships · CPC title

• G06V40/172

  Classification, e.g. identification · CPC title

• G06V10/454

  Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN] · CPC title

• G06T7/143

  involving probabilistic approaches, e.g. Markov random field [MRF] modelling · CPC title

• G06K9/00288 Primary

  Physics · mapped topic